Drug Database

# drugdatabase

## Introduction

In this project, I scrape the data from https://www.webmd.com/ website which contains information about health including drug, medicine and doctor for finding the most effective drug from customer review information. You can find the recommended way to cure your illness from this website. But in this project, I will focus on drug information of common condition as the website provide customer to review and rate the drug. So I will scrape this information to see which drug has the most effective and satisfaction to customer. Moreover, I will take a look at the effective of each drug form and what form is commonly use. 
Enjoy the programing. 

## Part 1 : Scraping Data

The first process is to scrape the drug information for each condition from the website. The information would consist of drug name, indication, type, form, average price, number of review, effective, ease of use, satisfaction from customer review. I use Beautiful Soup to read the content of html and collect all the links for all common condition which have 37 conditions. After that, I use for loop with selenium to open all links to find all drug links for each condition and then collect all information. Finally, I delete all Null values in data frame and save it as csv file.

## Part 2 : Cleaning Data

The next process is cleaning and preparing data for further analysis.  As the website did not contain the specific part of average price and form of the drug, I can take just paragraph of information. I have to extract information from the paragraph, I use tokenize function to get the sentence that contain ‘average’ word which contain average price and form in the sentence. After that, I split the sentence into word and get the information from it. The last cleaning process is to sum all drug review rating and find the average of drug that have the same name, same form for same condition as sometime the website contain the duplicate drug information. Finally, I have 685 drugs left for 37 conditions.

# Part 3 : Drug Database

As I would like to see the effective of drug by form, indication, type and condition, so I sum all data up and find the average by each group. After I got all the data, I create the drug Database. The drug database consists of 7 function.
First function, you can see the top drug for each condition by specific criteria.
Next function is to see the overall drug of each condition by criteria. 
Third function, it will show the boxplot of different form of drug for specific condition by criteria.
The next one is nearly the same as the third function, show overall rating the boxplot of all drug in database by different form.
The next two function is to see the different overall rating of OnLabel and OffLable, RX and OTC.
The last function is to see the what is the most commonly form of drug in the market by pie chart.

## Conclusion :

In conclusion, you can see from the pie chart, tablet is the most commonly sale in the market for relief 37 common condition. Next we take a look at price and effective of each form, we can see the effective rating are nearly the same, but liquid drink form is the most cheapest. So if you can, you would consider to buy liquid drug for saving your money. This boxplot show that price and effective of drug are not in the same trend which mean that expensive drug doesn’t mean the effective one. For the obvious example, meniere’s drug is really cheap but the effective rate is relatively high. The next graph btw on and off label, On-Label medications - FDA approved for the treatment of this condition. Off-Label medications - occasionally used but are not FDA approved for the treatment of this condition. Ease of use and effective average seem to be the same, but on-label drug are cheaper. For RX and OTC, Rx medications - require a prescription. OTC medications - may be purchased over the Counter. The average price of OTC drug are relative low compare to RX drug, and the effective of OTC drug is higher than RX. Moreover, OTC is easiler to find in the market, so you would like to buy this type of drug when you have some illness.