A quick introduction to Data Science

Data science is a multidisciplinary field that encompasses a diverse range of techniques, processes, and methodologies used to extract knowledge and insights from data. It combines elements of mathematics, statistics, computer science, domain expertise, and domain-specific knowledge to make informed decisions and predictions. In the modern age, where data has become a powerful resource, data science plays a pivotal role in transforming raw data into meaningful and actionable information. 

At its core, data science revolves around the concept of harnessing data to gain valuable insights and drive better decision-making. With the proliferation of technology and the internet, vast amounts of data are generated every day. This data comes from various sources such as social media interactions, online purchases, sensors, medical records, and more. However, raw data alone is of limited use; the real value lies in understanding and extracting patterns and trends hidden within this vast sea of information.

THE DATA SCIENCE WORKFLOW 

The workflow typically involves several key stages:

1. Data Collection: The first step is to gather data from diverse sources relevant to the problem at hand. This data can be structured (like databases) or unstructured (like text or images).

2. Data Cleaning and Preprocessing: Often, data may contain errors, missing values, or inconsistencies. Data scientists need to clean and preprocess the data to ensure its quality and prepare it for analysis.

3. Data Exploration and Visualization: In this stage, data scientists explore the data to uncover meaningful patterns, trends, and correlations. Visualization techniques are used to represent the data graphically, making it easier to understand and interpret.

4. Data Modeling: In this crucial phase, data scientists apply various mathematical and statistical techniques to build predictive models. These models can help in making predictions or classifications based on new data.

5. Model Training and Evaluation: The models are trained using historical data, and their performance is evaluated using metrics like accuracy, precision, recall, etc. This step helps in identifying the best performing model for the specific problem.

6. Deployment and Monitoring: Once a model is selected, it is deployed in real-world scenarios to make predictions or support decision making. Continuous monitoring ensures the model's performance remains optimal over time.

Read More

Typecasting

Typecasting is required to convert a value from one data type to another. You just saw an example in the last section where the value of price and quantity obtained by the user were in string format and were converted into integer and float values respectively with the help of typecasting.

Typecasting is a very simple and straightforward approach used to convert a variable from one data type to another data type.

Let’s say there are three variables num1, num2, and num3 as follows:

>>> num1 = 4

>>> type(num1)

<class 'int'>

>>> num2 = 3.6

>>> type(num2)

<class 'float'>

>>> num3 = 'dog'

>>> type(num3)

Now, let’s try to change the value of each variable to another type.

>>> num11 = float(4)

>>> num11

4.0

>>> type(num11)

<class 'float'>

>>> num22 = str(num2)

>>> type(num22)

<class 'str'>

>>> num33 = int(num3)

Traceback (most recent call last):

File "<pyshell#12>", line 1, in <module>

num33 = int(num3)

ValueError: invalid literal for int() with base 10: 'dog'

>>>

So, you see, you can convert numeric from one type to another and you can convert a numeric value to a string value but a string value cannot always be converted to a numeric value. Only if the string consists of a numeric value, can it be converted to a numeric data type. Following are some examples of typecasting a string into numeric values. Do have a look and don’t forget to experiment yourself.

a.

>>> str1 ='3.7'

>>> float(str1)

3.7

b.

>>> str2 = '3.7'

>>> int(str2)

Traceback (most recent call last):

File "<pyshell#16>", line 1, in <module>

int(str2)

ValueError: invalid literal for int() with base 10: '3.7'

c.

>>> str4 = '4.8'

>>> x = float(str4)

>>> int(x)

4

>>>

The problem in the example (b) can be resolved by first converting the string to float and then converting the float value to integer. This is shown in example(c) and another way is as follows:

>>> str2 = '3.7'

>>> int(float(str2))

3

>>>

Now that you have learned about typecasting, you would have also understood the importance of typecasting the user input into the right type before using it for any kind of arithmetic calculations.

Now Write a program to convert the value in grams (gm) into Kgs.

Answer:

gm = float(input("Enter value in Grams : "))

kg = gm/1000

print(gm,"grams = ",kg," Kgs.")

Output

Enter value in Grams : 8

8.0 grams = 0.008 Kgs.

>>>

Since a new value of 656 has been assigned to the reserved word ‘int’, you cannot use it for typecasting a float value to integer code. This code will generate a TypeError as follows:

Traceback (most recent call last):

File "<pyshell#3>", line 1, in <module>

int(num1)

TypeError: 'int' object is not callable

Read More

Possibilities of printing output

Notice the difference in print command in the case (I) and (II). Both print the same message.

1. The string message and variable values are separated by commas.

qty = int(input("How many apples do you have? : "))

price = float(input("What is the total cost? : "))

value_of_one = price/qty

print(qty,' apples cost ',price,' therefore one apple costs

',value_of_one,'.')

Output:

How many apples do you have? : 50

What is the total cost? : 250

50 apples cost 250.0 therefore one apple costs 5.0 .

>>>

2. The values of variables are inserted inside the string using format() method.

int(input("How many apples do you have? : "))

price = float(input("What is the total cost? : "))

value_of_one = price/qty

print('{} apples cost {} therefore one apple costs

{}.'.format(qty,price,value_of_one))

The first value in the parenthesis, that is, qty is embedded in the first curly bracket. The second value price is embedded in the second curly bracket and the last value, value_of_one is inserted in the last curly bracket.

Output:

How many apples do you have? : 20

What is the total cost? : 800

20 apples cost 800.0 therefore one apple costs 40.0.

>>>

You can change the order in which the variables are inserted in the string by numbering the curly brackets.

3.

qty = int(input("How many apples do you have? : "))

price = float(input("What is the total cost? : "))

value_of_one = price/qty

print('{2} will be the cost of one apple if {0} apple cost

{1}.'.format(qty,price,value_of_one))

Output:

How many apples do you have? : 10

What is the total cost? : 500

50.0 will be the cost of one apple if 10 apple cost 500.0.

>>>

By default whitespace acts as a separator between the arguments to print() function in Python.

print('What would you like to have?')

print('Rice','lentils','veggies','?')

Output:

What would you like to have?

Rice lentils veggies?

If you want any other separator instead of whitespace, you can mention that in the print function using sep parameter as follows:

print('What would you like to have?')

print('Rice','lentils','veggies','?',sep='/')

Output:

What would you like to have?

Rice/lentils/veggies/?

Read More

Getting the user input and displaying output

Python has an in-built input() function that has an optional argument which is a prompt string. A prompt string can be a message for the user on the prompt that tells him/her about the value that he/she must provide.

During the execution of a program, if the interpreter encounters the input() method, it halts the program till the user provides input and hits the enter key.

Syntax

>>>input([prompt])

Look at the input() method shown as follows:

>>>input('Hi there! Do You live in New York?(Yes/No) :')

Here,

[prompt] = 'Hi there! Do You live in New York?(Yes/No) :'

You will notice that the cursor in front of the prompt string keeps blinking till the user enters a response, and hits the enter key. Till that happens the program execution is put to a halt.

The purpose of this method is to take input from the user and assign it to a variable. The following figure displays the user providing an input value ’Yes’ and pressing the enter key.

>>>user_input = input('Hi there! Do You live in New York?(Yes/No) :') 

Hi there! Do You live in New York?(Yes/No) : Yes

>>> print('the user says: ',user_input)

the user says : Yes

>>>

So, in the preceding code, the input() method is used to capture the response of the user which is a Yes and then the response is assigned to a variable by the name user_input,. Look at the print() method. It concatenates the string ‘The user says : ’ with the value stored in user_input variable and prints the value.

Let’s check the type of value obtained from the input() method (in other words, value of user_input).

>>>user_input = input('Hi there! Do You live in New York? (Yes/No) :')

Hi there! Do You live in New York?(Yes/No) :Yes

>>> type(user_input)

<class 'str'>

>>>

You may wonder what’s the need to check the type of user_input when clearly, the user is providing a string as input? You will understand the importance in the next example. 

Let’s try to use input() method to get user input to perform some arithmetic function. Look at the following code :

qty = input("How many apples do you have? : ")

price = input("What is the total cost? : ")

value_of_one = price/qty

print(value_of_one)

The output will be as follows:

How many apples do you have? : 80

What is the total cost? : 160

Also notice the error message: ‘unsupported operand type(s) for /: 'str' and 'str'’. This error was generated when an attempt was made to divide the variable price with the variable qty. The message is conveying that you made an attempt to divide a string variable with another string variable.

This means that although the user entered a numeric value, Python is treating it as a string value. This is because the input function always returns a string value. No matter what value is provided by the user, the input function will always treat it like a string.

So, in order to divide the price by quantity, you must convert both the values into the right data type.

qty = int(input("How many apples do you have? : "))

price = float(input("What is the total cost? : "))

value_of_one = price/qty

print(value_of_one)

The output will be as follows:

How many apples do you have? : 20

What is the total cost? : 256.50

12.825

In the preceding example, look at the expression int(input("How many apples do you have? : ")). This expression typecasts the value received by the user into an integer.

Read More

Date/Time module

Following is the code to retrieve current date and time

import datetime

print(datetime.datetime.today())

Or

import datetime

print(datetime.datetime.now())

Code to display current hour of the day

from datetime import *

print(getattr(datetime.today(),'hour'))

Code to print current local date

In order to display the present local date we can use of today() method which is defined in the date class.

import datetime

date_object = datetime.date.today()

print(date_object)

Output:

2020-07-22

Function that can be used to see which attributes belong to the datetime module

dir(datetime)

Important classes of the datetime module

1. Date

2. Time

3. Datetime

4. Timedelta

Now what would be the output of the following code:

import datetime

date1 = datetime.date(2020, 7, 23)

print(date1)

Well the Answer is:

2020-07-23

The date() function is actually a constructor of the date class, and it take three arguments: Year, month, and date. This code can also be written as:

from datetime import date

date1 = date(2020, 7, 23)

print(date1)

Now try to write the code to display present date. It should be something like:

from datetime import date

date_today = date.today()

print("Today's date is", date_today)

Output:

Today's date is 2020-07-22

What if someone asks you to print today’s month, year and date separately?

You can print using the following code-

from datetime import date

date_today = date.today()

print("Today's day:", date_today.day, end= " ")

print("Today's month:", date_today.month, end= " ")

print("Today's year:", date_today.year)

Run this code and you should get the output as current month, year and date.

Read More

The random numbers

To work with random numbers you will have to import random module.

The following functions can be used:

import random

#Random choice

print (" Working with Random Choice")

seq=[8,3,5,2,1,90,45,23,12,54]

print ("select randomly from ", seq," : ",random.choice(seq))

print("*******************")

#randomly select from a range

print ("randomly generate a number between 1 and 10 :

",random.randrange(1, 10))

print("*******************")

#random()

print ("randomly display a float value between 0 and 1 :

",random.random())

print("* * * * * *")

#shuffle elements of a list or a tuple

seq=[1,32,14,65,6,75]

print("shuffle ",seq,"to produce : ",random.shuffle(seq))

#uniform function to generate a random float number between two numbers

print ("randomly display a float value between 65 and 71 :

",random.uniform(65,71))

OUTPUT

Working with Random Choice

select randomly from [8, 3, 5, 2, 1, 90, 45, 23, 12, 54] : 2

*******************

randomly generate a number between 1 and 10 : 8

*******************

randomly display a float value between 0 and 1 :

0.3339711273144338

* * * * * *

shuffle [1, 32, 14, 75, 65, 6] to produce : None

randomly display a float value between 65 and 71 :

65.9247420528493 

Read More

The math module - trigonometric functions

 Some of the trigonometric functions defined in the math module are as follows:

import math

# calculate arc tangent in radians

print ("atan(0) : ",math.atan(0))

print("**************")

# cosine of x

print ("cos(90) : ",math.cos(0))

print("**************")

# calculate hypotenuse

print ("hypot(3,6) : ",math.hypot(3,6))

print("**************")

# calculates sine of x

print ("sin(0) : ", math.sin(0))

print("**************")

# calculates tangent of x

print ("tan(0) : ",math.tan(0))

print("**************")

# converts radians to degree

print ("degrees(0.45) : ",math.degrees(0.45))

print("**************")

# converts degrees to radians

print ("radians(0) : ",math.radians(0))

OUTPUT

atan(0) : 0.0

**************

cos(90) : 1.0

**************

hypot(3,6) : 6.708203932499369

**************

sin(0) : 0.0

**************

tan(0) : 0.0

**************

degrees(0.45) : 25.783100780887047

**************

radians(0) : 0.0

>>>

Read More

The math module - Mathematical functions

Mathematical functions are defined in the math module. You will have to import the math module to get access to all its functions.

import math

#ceiling value

a = -52.3

print ("math ceil for ",a, " is : ", math.ceil(a))

print("********************")

#exponential value

a = 2

print("exponential value for ", a, " is: ",math.exp(2))

print("********************")

#absolute value of x

a = -98.4

print ("absolute value of ",a," is: ",abs(a))

print("********************")

#floor values

a = -98.4

print ("floor value for ",a," is: ", math.floor(a))

print("********************")

# log(x)

a = 10

print ("log value for ",a," is : ", math.log(a))

print("********************")

# log10(x)

a = 56

print ("log to the base 10 for ",a," is : ",math.log10(a))

print("********************")

# to the power of

a = 2

b = 3

print (a," to the power of ",b," is : ",math.pow(2,3))

print("********************")

# square root

a = 2

print("sqaure root")

print ("Square root of ",a," is : ", math.sqrt(25))

print("********************")

OUTPUT

math ceil for -52.3 is : -52

********************

exponential value for 2 is: 7.38905609893065

********************

absolute value of -98.4 is: 98.4

********************

floor value for -98.4 is: -99

********************

log value for 10 is : 2.302585092994046

********************

log to the base 10 for 56 is : 1.7481880270062005

********************

2 to the power of 3 is : 8.0

********************

sqaure root

Square root of 2 is : 5.0

********************

Read More

Strings in-built or predefined methods

All objects in Python whether strings, tuples, lists, and so on have some inbuilt methods that they are associated with. People often get confused between a method and a function and many think that it is one and the same thing but the fact is that in Python there is a difference between the methods and functions. Functions have parenthesis and arguments and when these functions get associated with an object they become a method.

Few interesting features:

Whenever, you type a dot ‘.’ operator in front of an object, the idle displays all the methods that can be associated with it. 

You will initially work with the following:

help()

find()

upper()

lower()

strip()

replace()

split()

join()

in and not in (These are membership operator and not methods)

endswith()

You can use dir(str) method to see all the methods associated with string objects.

1. help()

To find complete information about any method use help() function.

The help() function applies to all Python objects.

2. find() 

The find() method will return the lowest index in the string where the desired substring exists.

S.find(sub[, start[, end]])

Where, S is the string, sub is the substring that you are looking for. Arguments in the square brackets are not mandatory.

>>>#find()

>>> x = 'Last Section of the Chapter'

>>> x.find('ast')

1

>>>

3. upper()

This method converts the entire string into upper case.

>>>#upper()

>>> x = 'Last Section of the Chapter'

>>> x.upper()

'LAST SECTION OF THE CHAPTER'

>>>

4. lower()

This method converts entire string into lowercase.

>>>#lower()

>>> x = 'Last Section of the Chapter'

>>> x.lower()

'last section of the chapter'

>>>

5. strip()

This method removes whitespaces or a particular character from a string.

>>> #strip()

>>> money = '$100'

>>> money.strip('$')

'100'

6. replace()

This method removes a character or substring in a string with some other character or string.

>>> #replace()

>>> str1 = 'Happy Chrsitmas'

>>> str1.replace('Happy','Merry')

'Merry Chrsitmas'

>>>

7. split()

This splits a string into a list data type based on the character that is passed as an argument.

>>> #split()

>>> x = 'Last Section of the Chapter'

>>> x.split()

['Last', 'Section', 'of', 'the', 'Chapter']

>>> ip ='222:222:0:02'

>>> ip.split(':')

['222', '222', '0', '02']

>>>

8. join()

Opposite of string.

>>> #join()

>>> student = ['Alex','32','Physics Major', 'Baseball']

>>> ('|').join(student)

'Alex|32|Physics Major|Baseball'

>>>

9. in and not in

Checks if a substring is a part of the string or not.

>>> #in

>>> x = 'Merry Christmas'

>>> 'Meery' in x

False

>>> 'Merry' in x

True

>>> #not in

>>> 'year' not in x

True

>>> 'Christmas' not in x

False

>>>

10. endswith()

This method is used to check if the string ends with a particular string or not.

>>> #endswith()

>>> x = 'Merry Christmas'

>>> x.endswith()

>>> x.endswith('as')

True

>>>

Read More

Converting Unstructured Data to Structured Form

As a data scientist, you not only need to fetch the data but also analyze it. Storing the data in a structured form simplifies this task. In this section, we will learn how to convert the data fetched from MongoDB into a structured format.

Storing into a Dataframe

The find function returns a dictionary from a MongoDB collection. You can directly insert it into a dataframe. First, let’s fetch 100 MongoDB documents and then we will store these documents into a dataframe:

import pandas as pd

samples=table.find().sort("_id",pymongo.DESCENDING)[:100]

df=pd.DataFrame(samples)

df.head()

The readability of this dataframe is far better than that of the default format returned by the function.

Writing to a File

Pandas dataframes can directly be exported into CSV, Excel or SQL. Let us try to store this data to a CSV file:

df.to_csv('StructuredData.csv',index=False)

Similarly, you can use the to_sql function to export the data into a SQL database.

Read More

Creating Database and Collection

The creation of any database and collection is a very simple process in MongoDB. You can use the syntax of retrieval to do this. If you try to access a database which doesn’t exist, MongoDB will create it for you.

Let’s create a database and a collection:

mydb=client.testDB

mycoll=mydb.testColl

The MongoDB database has been created here but if we run list_database_names, this database will not be listed. MongoDB doesn’t show empty databases. So, we will have to insert something there. Let’s insert a document in the MongoDB collection:

testInsert=mycoll.insert_one({"country":'India'}).inserted_id

client.list_database_names()

Now we can see that our database is available in the list of MongoDB databases.

Read More