Bhishan Bhandari / Product and Engineering

Python and Excel

2018-12-09T00:00:00+00:00

I intend to host a set of examples on using python to interact and work with excel files. This article in particular will use openpyxl module in python throughout the examples.

Installing openpyxl

I am using python3 throughout the examples, however it should work similarly with python2. As a good development practice, we should segregate dependencies between various projects and therefore is advised to use virtual environment or some sort.

python3 -m venv excelenv
source excelenv/bin/activate
pip install openpyxl

Open an existing document in python

>>> import openpyxl
>>> wb = openpyxl.load_workbook('example.xlsx')
>>> wb

>>>

Selecting sheets from the workbook/document

>>> wb.get_sheet_names()
['Sheet1']
>>>
>>> sheet = wb.get_sheet_by_name("Sheet1")
>>> sheet

>>>
>>> sheet.max_column
3
>>> sheet.max_row
3
>>> sheet.min_row
1
>>> sheet.min_column
2
>>>
>>> sheet['A1'].value
'thetaranights.com'
>>> sheet['B1'].value
102345
>>> sheet['A2'].value
'thetaranights.com'
>>> sheet['B2'].value
123443
>>>

When the cell being accessed doesn’t have a value

>>> a = sheet['A4'].value
>>> a
>>> type(a)

>>>

OpenPyXL will automatically interpret the types of the values in the cells of the sheet and return them as an object of that type. string , int, dates, etc.

>>> type(sheet['A3'].value)

>>> type(sheet['B3'].value)

Accessing cell values using row, column directive

Although we can access the values of the cell using alphabetic letters directive, we can also access them using row number and column numbers. An excel row and column starts at 1, not 0.

>>> sheet.cell(row=0, column=0)
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/bhishan-1504/excelinpython/excelenv/lib/python3.6/site-packages/openpyxl/worksheet/worksheet.py", line 296, in cell
    raise ValueError("Row or column values must be at least 1")
ValueError: Row or column values must be at least 1
>>>

>>> sheet.cell(row=1, column=1)

>>> sheet.cell(row=1, column=1).value
'thetaranights.com'
>>>

Reading values from excel sheet

>>> wb = openpyxl.load_workbook('example.xlsx')
>>> sheet = wb.get_sheet_by_name("Sheet1")
>>> for each_row in range(sheet.min_row, sheet.max_row + 1):
...     for each_col in range(sheet.min_column, sheet.max_column + 1):
...         print(sheet.cell(row=each_row, column=each_col).value)
...
thetaranights.com
102345
thetaranights.com
123443
thetaranights.com
102234
>>>

Creating excel document

>>> import openpyxl
>>> wb = openpyxl.Workbook()
>>> wb.get_sheet_names()
['Sheet']
>>> sheet = wb.get_sheet_by_name('Sheet')
>>> sheet

>>> sheet.title = 'Custom Created worksheet'
>>> wb.get_sheet_names()
['Custom Created worksheet']
>>> sheet

>>> sheet['A1'] = 'thetaranights.com'
>>> sheet['A1'].value
'thetaranights.com'
>>>
>>> sheet.append(['thetaranights.com', '102948']) # adding rows to sheet
>>> wb.save('createdbyopenpyxl.xlsx')
>>>

Idiomatic Python - Use of Falsy and Truthy Concepts

2018-12-05T00:00:00+00:00

Out of many, one reason for python’s popularity is the readability. Python has code style guidelines and idioms and these allow future readers of the code to comprehend to the intentions of it. It is highly important that the code is readable and concise. One such important tip is to use falsy and truthy concepts.

It should be at our best interest to avoid direct comparison to True, False, None. As such we should be well known about truthy and falsy concepts.

Truthy refers to the values that shall always be considered true. Similarly falsy refers to the values that shall always be considered false.

An empty sequence such as an empty list [], empty dictionaries, 0 for numeric, None are considered false values or falsy. Almost anything excluding the earlier mentioned are considered truthy.

An example of a code snippet that is considered bad:

i = 0
if i == 0:
    foo()
    # do something
else:
    bar()
    # do something else

x = True
if x == True:
    foo()
    # do something
else:
    bar()
    # do something else

numbers_list = [1, 2, 3, 4]
if len(numbers_list) > 0:
    foo()
    # do something

An example of code snippet that is considered relatively wise:

i = 0
if not i:
    foo()
    # do something
else:
    bar()
    # do something else

x = True
if x:
    foo()
    # do something
else:
    bar()
    # do something else

numbers_list = [1, 2, 3, 4]
if numbers_list:
    foo()
    # do something

Basic File Operations - Golang

2018-12-05T00:00:00+00:00

One of the most basic task when working on a server is the ability to effectively operate with the files and file system. Like many languages, Golang has convenient methods to work with files. The intentions of this post is to host a minimalist set of examples on working with files using Golang.

Creating an empty file

package main

import (
  "log"
  "os"
)

func createFile(){
  newFile, err := os.Create("testfile.txt")
  if err != nil{
     log.Fatal(err)
  }
  defer newFile.Close()

  log.Println(newFile)
  // operations on file follows
}


func main(){
  createFile()
}

Reading a File

func readFile(){
  fileReader, err := os.Open("test.txt")
  if err != nil{
     log.Fatal(err)
  }
   defer fileReader.Close()
  content, err := ioutil.ReadAll(fileReader)
  if err != nil{
     log.Fatal(err)
  }
  fmt.Println("Number of bytes read", len(content))
  fmt.Printf("Data as string : %s", content)


}

Writing to a file

func writeToFile(){
  err := ioutil.WriteFile("testwrite.txt", []byte("Dumping bytes to a file\n"), 0666)
  if err != nil{
     log.Fatal(err)
  }
}

Truncate File

func TruncateFile(){
  err := os.Truncate("test.txt", 100)
  if err != nil{
     log.Fatal(err)
  }else{
     fmt.Print("File Truncated")
  }

}

File Information

func FileInformation(){
  fileInfo, err := os.Stat("test.txt")
  if err != nil {
     log.Fatal(err)
  }
  fmt.Println("File name:", fileInfo.Name())
  fmt.Println("Size in bytes:", fileInfo.Size())
  fmt.Println("Permissions:", fileInfo.Mode())
}

Check if file exists or not

package main

import (
    "os"
    "fmt"
)

func main(){
    if _, err := os.Stat("test.txt"); os.IsNotExist(err){
        fmt.Println("The file does not exist")
    return
    }
    fmt.Println("The file exists!")
    //perform operations on the file/file contents
}

Move/Rename File

package main
 
import (
    "log"
    "os"
)
 
func main() {
    currentFileLocation := "/home/bhishan-1504/golangfilehandling/test.txt"
    toMoveLocation := "/home/bhishan-1504/golangrestapi/test.txt"
    err := os.Rename(currentFileLocation, toMoveLocation)
    if err != nil {
        log.Fatal(err)
    }
}

Copy File

package main

import (
  "io"
  "log"
  "os"
  "fmt"
)

func main(){
  source, err := os.Open("test.txt")
  if err != nil{
    log.Fatal(err)
  }
  defer source.Close()

  sourceCopied, err := os.OpenFile("copiedtest.txt", os.O_RDWR|os.O_CREATE, 0666)
  //sourceCopied, err := os.Create("copiedtest.txt")
  if err != nil{
    log.Fatal(err)
  }
  defer sourceCopied.Close()

  _, err = io.Copy(sourceCopied, source)
  if err != nil{
    log.Fatal(err)
  }

 
  fmt.Println("Copied!")
}

Going serverless with Chalice and AWS lambda

2018-12-05T00:00:00+00:00

The intentions of this post is to host a simple example of chalice from AWS that allows serverless API creation with the use of AWS lambda. You also get auto-generation of IAM policy making it faster to deploy web applications. Chalice expects to pick the AWS credentials from ~/.aws/config

Prerequisites for chalice[AWS Credentials]

If you’ve used AWS API or boto/boto3 for python, you’ve probably already added the credentials in place. Otherwise you could do it as below:

mkdir ~/.aws
touch ~/.aws/config

Contents of the config file

[default]
aws_access_key_id = YOUR_ACCESS_KEY_HERE
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
region = YOUR_REGION

Installing Chalice

As a good development practice, we should segregate dependencies between various projects and therefore is advised to use virtual environment or some sort.

$ python3 -m venv serverlessenv
$ source serverlessenv/bin/activate

(serverlessenv)$ pip install chalice

Creating simple API with chalice

Chalice comes with command line tool. See what commands are available

(serverlessenv)$ chalice

Usage: chalice [OPTIONS] COMMAND [ARGS]...

Options:
--version Show the version and exit.
--project-dir TEXT The project directory. Defaults to CWD
--debug / --no-debug Print debug logs to stderr.
--help Show this message and exit.

Commands:
delete
deploy
gen-policy
generate-pipeline Generate a cloudformation template for a...
generate-sdk
invoke Invoke the deployed lambda function NAME.
local
logs
new-project
package
url

(serverlessenv)$ chalice new-project

___ _ _ _ _ ___ ___ ___
/ __|| || | /_\ | | |_ _|/ __|| __|
| (__ | __ | / _ \ | |__ | || (__ | _|
\___||_||_|/_/ \_\|____||___|\___||___|

The python serverless microframework for AWS allows
you to quickly create and deploy applications using
Amazon API Gateway and AWS Lambda.

Please enter the project name: basichelloworld

(serverlessenv)$ cd basichelloworld/

(serverlessenv)~/basichelloworld$ ls -a

app.py .chalice .gitignore requirements.txt

Chalice is very simple, somewhat similar to Flask if you come from there.

Auto-generated app.py

from chalice import Chalice

app = Chalice(app_name='basichelloworld')


@app.route('/')
def index():
    return {'hello': 'world'}


# The view function above will return {"hello": "world"}
# whenever you make an HTTP GET request to '/'.
#
# Here are a few more examples:
#
# @app.route('/hello/{name}')
# def hello_name(name):
#    # '/hello/james' -> {"hello": "james"}
#    return {'hello': name}
#
# @app.route('/users', methods=['POST'])
# def create_user():
#     # This is the JSON body the user sent in their POST request.
#     user_as_json = app.current_request.json_body
#     # We'll echo the json body back to the user in a 'user' key.
#     return {'user': user_as_json}
#
# See the README documentation for more examples.
#

Running the API locally

(serverlessenv)~/basichelloworld$ chalice local
Serving on 127.0.0.1:8000

(serverlessenv)~/basichelloworld$ curl -X GET http://127.0.0.1:8000
{"hello": "world"}

Going serverless with chalice through AWS lambda

(serverlessenv)~/basichelloworld$ chalice deploy
Creating deployment package.
Updating policy for IAM role: basichelloworld
Creating lambda function: basichelloworld
Creating Rest API
Resources deployed:
- Lambda ARN: arn:aws:lambda:us-east-1:9582857991:function:basichelloworld
- Rest API URL: https://fxcdyzuitc.execute-api.us-east-1.amazonaws.com/api/

(serverlessenv)~/basichelloworld$ curl -X GET https://fxcdyzuitc.execute-api.us-east-1.amazonaws.com/api/
{"hello": "world"}

Top 10 Most Subscribed Youtube Channel Visualization[2013-2018]

2018-11-24T00:00:00+00:00

Motion Visualization is intriguing and explains a lot in itself. I have put together an example visualization of the Top 10 Youtubers(by subscribers) from November 2013 to November 2018. I am not very informed of whether or not flash based visualizations are still relevant, but I found Google Motion Chart to have explained very well the data set I’ve taken into consideration. As always the codes remain public so people can take ideas from it and produce an even better visualizations for themselves.

It is especially interesting to produce a motion chart for data points where entities have varying rates of growth overtime. I had difficulties being able to view the visualizations on some versions of chrome although I had flash installed. Make sure you have installed adobe flash in-order to view the visualizations. Here is a link to the visualization.

I’ve also prepared a video from the visualization for the ones having trouble viewing the viz. You should watch it anyways, it’s fun.

https://youtu.be/vMeKjwV15K8

The code below contains only a few portion of the dataset to make it much readable.

html

Python, Boto and AWS EC2

2018-09-16T00:00:00+00:00

Most if not all software companies have adopted to cloud infrastructure and services. AWS in particular is very popular amongst all. The intentions of this post is to host a few examples on using boto to make use of one of the services available on AWS i.e EC2. It is more likely than not to have need of a mechanism to programatically fire up a few instances, shut them down, filter instances and send remote commands to it to say the least.

Filter instances based on tag names from the AWS inventory

EC2 instances on AWS can have as many tag names key: value as required for purposes like identifying an instance or a set of instances. Also when the instance you are working on quite frequently needs to shut down and boot over again and you haven’t implemented elastic IP, you are bound to changes in the public IP address. Although you could argue to use private IP to filter an instance, it isn’t very effective when you have a lot of instances(>100).

Boto2

import boto.ec2

conn = boto.ec2.connect_to_region('us-east-1', aws_access_key_id='aws_access_id', aws_secret_access_key='aws_secret')
reservations = conn.get_all_instances(filters={'tagName' : 'value'})
public_ips = [each_instance.ip_address for r in reservations for each_instance in r.instances]
# each_instance.private_ip_address  to get the private ip address of the instance

Boto3

import boto3
session = boto3.session.Session(aws_access_key_id=aws_access_id,
                                aws_secret_access_key=aws_secret,
                                region_name='us-east-1')
 
ec2 = session.resource('ec2')
instances = ec2.instances.filter(
    Filters=[{'Name':'tag:purpose', 'Values':['intelligence']}
])
public_ips = [each_instance.public_ip_address for each_instance in instances]
# each_instance.private_ip_address to get the private ip address of the instance

Boot/Shutdown an instance/instances from the AWS inventory

Using boto, you can boot/shutdown/terminate instances.

Boto2

def start_stop_terminate_instance(instance_ids, conn, action='start'):
    if action == 'start':
        conn.start_instances(instance_ids=instance_ids)
    elif action == 'stop':
        conn.stop_instances(instance_ids=instance_ids)
    elif action == 'terminate':
        conn.terminate_instances(instance_ids=ids)

Boto3

def start_stop_terminate_instance(instance_ids, conn, action='start'):
    if action == 'start':
        conn.instances.filter(InstanceIds=instance_ids).start()
    elif action == 'stop':
        conn.instances.filter(InstanceIds=instance_ids).stop()
    elif action == 'terminate':
        conn.instances.filter(InstanceIds=instance_ids).terminate()

Create Instances based on various metrics

Boto makes use of the AWS APIs that also allows creating instances. An EC2 instance can have various properties. The most common is the type of the instance. Types are generally a grouping of instances based on metrics such as power, performance, bandwidth. Commonly used types for general purpose are t2, m4, m3. C5, c4, c3 are compute optimized instances. For a process/application more leaned towards in-memory activities, you’d use x1, r4, r3. There are other types too but the above mentioned are quite common in use. The other properties of an instance are instance id, the memory size (micro, nano, small, large, xlarge, 2xlarge, 4xlarge, 8xlarge, 10xlarge.), the key pair to make a secured connection to the instance, tag names, display names, security groups, attached storage id, etc. Using boto we can create an instance or multiple instances based on the above mentioned parameters.

Boto2

import boto.ec2
conn = boto.ec2.connect_to_region('us-east-1', aws_access_key_id='aws_access_id', aws_secret_access_key='aws_secret')
conn.run_instances(
    'ami-ag139jf',
    min_count=10, 
    max_count=100,
    key_name='myKey',
    instance_type='t2.small',
    security_groups=['sg-4512']
)

Boto3

import boto3
session = boto3.session.Session(aws_access_key_id='aws_access_id',
                                aws_secret_access_key='aws_secret',
                                region_name='us-east-1')
 
ec2 = session.resource('ec2')
ec2.create_instances(
    ImageId='ami-ag139jf', 
    MinCount=10, 
    MaxCount=100, 
    InstanceType='t2.small',
    KeyName='myKey',
    SecurityGroups=['sg-4512']
)

Send remote commands to an EC2 instance

Paramiko can be used for connecting to a remote instance and sending commands to be executed and get the standard output/error to act accordingly.

import paramiko

key = paramiko.RSAKey.from_private_key_file(path_to_pem_file)
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

# Connect to the instance
try:
    # using username, public ip address and the pem file, create connection to the instance
    client.connect(hostname=instance_ip, username="ubuntu", pkey=key)

    # Execute command remotely.
    stdin, stdout, stderr = client.exec_command("ls -l")
    print stdout.read()
    client.close()

except Exception, e:
    print e

Web Scraping with Golang and Goquery

2018-09-13T00:00:00+00:00

Web Scraping can be beneficial to individuals and companies. The intentions of this post is to host a set of examples on Web Scraping using Golang and goquery. I will be using github’s trending page https://github.com/trending throughout this post for the examples, especially because it best suits for applying various goquery methods. There are two other versions of this article which replicates the same set of examples in Python and NodeJS.

Installation

go get github.com/PuerkitoBio/goquery

Get html of a page

package main
import (
    "log"
    "io"
    "os"
    "net/http"
)

func ScrapeHTML(){
    resp, err := http.Get("https://github.com/trending")
    if err != nil{
        log.Fatal(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200{
        log.Fatalf("status code error: %d %s", resp.StatusCode, resp.Status)
    }
    io.Copy(os.Stdout, resp.Body)    
}

func main(){
    ScrapeHTML()
}

Using goquery(golang library) to get title from a page

package main
import (
    "fmt"
    "log"
    "net/http"
    "github.com/PuerkitoBio/goquery"
)

func ScrapeHTML(){
    resp, err := http.Get("https://github.com/trending")
    if err != nil{
        log.Fatal(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200{
        log.Fatalf("status code error: %d %s", resp.StatusCode, resp.Status)
    }

    doc, err := goquery.NewDocumentFromReader(resp.Body)
  if err != nil {
        log.Fatal(err)
    }
    fmt.Println(doc.Find("title").Text())
   
}

func main(){
    ScrapeHTML()
}

Output

$ go run example.go
Trending repositories on GitHub today · GitHub

Using goquery, Find single element by tag name, find multiple elements by tag name

package main
import (
    "fmt"
    "log"
    "strings"
    "net/http"
    "github.com/PuerkitoBio/goquery"
)


func scrapeUsingTagNames(){
    resp, err := http.Get("https://github.com/trending")
    if err != nil{
        log.Fatal(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200{
        log.Fatalf("Status code error: %d %s", resp.StatusCode, resp.Status)
    }
    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil{
        log.Fatal(err)
    }
    fmt.Println(doc.Find("title").Text())

    doc.Find("ol li").Each(func(i int, s *goquery.Selection){
        fmt.Println(strings.TrimSpace(s.Find("h3").Text()))
    })
}

func main(){
    scrapeUsingTagNames()
}

Output

$ go run example.go
Trending repositories on GitHub today · GitHubAsset 1Asset 1
you-dont-need / You-Dont-Need-Momentjs
ripienaar / free-for-dev
Nozbe / WatermelonDB
cjbarber / ToolsOfTheTrade
byoungd / English-level-up-tips-for-Chinese
TheAlgorithms / Python
thedaviddias / Front-End-Checklist
zziz / pwc
dawnlabs / carbon
CyC2018 / CS-Notes
Avik-Jain / 100-Days-Of-ML-Code
donnemartin / system-design-primer
mariusandra / pigeon-maps
Snailclimb / JavaGuide
JavaNoober / BackgroundLibrary
crossoverJie / JCSprout
Microsoft / nni
PansonPanson / Java-Notes
date-fns / date-fns
sindresorhus / ky
mciastek / sal
rwv / chinese-dos-games
vuejs / vue
GoogleCloudPlatform / open-match
lin-xin / vue-manage-system

Getting Attributes of an element

package main
import (
    "fmt"
    "log"
    "net/http"
    "github.com/PuerkitoBio/goquery"
)

func scrapeAttributes(){
    resp, err := http.Get("https://github.com/trending")
    if err != nil{
        log.Fatal(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200{
        log.Fatalf("Status code error: %d %s", resp.StatusCode, resp.Status)
    }
    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil{
        log.Fatal(err)
    }
    fmt.Println(doc.Find("title").Text())

    doc.Find("ol li").Each(func(i int, s *goquery.Selection){
        href, has_attr := s.Find("a").First().Attr("href")
        if has_attr{
            fmt.Println("https://github.com" + href)
        }

    })
}

func main(){
    scrapeAttribtutes()
}

Output

$ go run example.go
Trending repositories on GitHub today · GitHubAsset 1Asset 1
https://github.com/you-dont-need/You-Dont-Need-Momentjs
https://github.com/ripienaar/free-for-dev
https://github.com/Nozbe/WatermelonDB
https://github.com/cjbarber/ToolsOfTheTrade
https://github.com/byoungd/English-level-up-tips-for-Chinese
https://github.com/TheAlgorithms/Python
https://github.com/thedaviddias/Front-End-Checklist
https://github.com/zziz/pwc
https://github.com/dawnlabs/carbon
https://github.com/CyC2018/CS-Notes
https://github.com/Avik-Jain/100-Days-Of-ML-Code
https://github.com/donnemartin/system-design-primer
https://github.com/mariusandra/pigeon-maps
https://github.com/Snailclimb/JavaGuide
https://github.com/JavaNoober/BackgroundLibrary
https://github.com/crossoverJie/JCSprout
https://github.com/Microsoft/nni
https://github.com/PansonPanson/Java-Notes
https://github.com/date-fns/date-fns
https://github.com/sindresorhus/ky
https://github.com/mciastek/sal
https://github.com/rwv/chinese-dos-games
https://github.com/vuejs/vue
https://github.com/GoogleCloudPlatform/open-match
https://github.com/lin-xin/vue-manage-system

Using class name or other attributes to get element

package main
import (
    "fmt"
    "log"
    "strings"
    "net/http"
    "github.com/PuerkitoBio/goquery"
)
func scrapeViaClassName(){
    resp, err := http.Get("https://github.com/trending")
    if err != nil{
        log.Fatal(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200{
        log.Fatalf("Status code error: %d %s", resp.StatusCode, resp.Status)
    }
    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil{
        log.Fatal(err)
    }
    fmt.Println(doc.Find("title").Text())

    doc.Find("ol li").Each(func(i int, s *goquery.Selection){
        fmt.Println(strings.TrimSpace(s.Find(".float-sm-right").Text()))
    })
}


func main(){
    scrapeViaClassName()
}

Output

$ go run example.go
Trending repositories on GitHub today · GitHub
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today

Navigate childrens from an element

package main
import (
    "fmt"
    "log"
    "strings"
    "net/http"
    "github.com/PuerkitoBio/goquery"
)
func navigateChildrens(){
    resp, err := http.Get("https://github.com/trending")
    if err != nil{
        log.Fatal(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200{
        log.Fatalf("Status code error: %d %s", resp.StatusCode, resp.Status)
    }
    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil{
        log.Fatal(err)
    }
    fmt.Println(doc.Find("title").Text())
    olSelection := doc.Find("ol")
    olSelection.Children().Each(func(i int, s *goquery.Selection){ // using .Children() on the ol selection to get all li
        fmt.Println(strings.TrimSpace(s.Find("h3").Text()))
    })
}


func main(){
    navigateChildrens()
}

Output

$ go run example.go
Trending repositories on GitHub today · GitHub
you-dont-need / You-Dont-Need-Momentjs
ripienaar / free-for-dev
Nozbe / WatermelonDB
cjbarber / ToolsOfTheTrade
byoungd / English-level-up-tips-for-Chinese
TheAlgorithms / Python
thedaviddias / Front-End-Checklist
zziz / pwc
dawnlabs / carbon
CyC2018 / CS-Notes
Avik-Jain / 100-Days-Of-ML-Code
donnemartin / system-design-primer
mariusandra / pigeon-maps
Snailclimb / JavaGuide
JavaNoober / BackgroundLibrary
crossoverJie / JCSprout
Microsoft / nni
PansonPanson / Java-Notes
date-fns / date-fns
sindresorhus / ky
mciastek / sal
rwv / chinese-dos-games
vuejs / vue
GoogleCloudPlatform / open-match
lin-xin / vue-manage-system

The .children will only return the immediate childrens of the parent element.

Navigating previous and next siblings of an element

package main
import (
    "fmt"
    "log"
    "strings"
   "net/http"
    "github.com/PuerkitoBio/goquery"
)

func navigateSiblings(){
    resp, err := http.Get("https://github.com/trending")
    if err != nil{
        log.Fatal(err)
    }
    defer resp.Body.Close()
    if resp.StatusCode != 200{
        log.Fatalf("Status code error: %d %s", resp.StatusCode, resp.Status)
    }
    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil{
        log.Fatal(err)
    }
    fmt.Println(doc.Find("title").Text())
    liSelection := doc.Find("ol li")
    fifthElement := liSelection.Eq(4) // using Eq() and passing the index we can navigate to the element with given index
    fmt.Println(strings.TrimSpace(fifthElement.Find("h3").Text()))
    fourthElement := fifthElement.Prev()
    fmt.Println(strings.TrimSpace(fourthElement.Find("h3").Text()))
    sixthElement := fifthElement.Next()
    fmt.Println(strings.TrimSpace(sixthElement.Find("h3").Text()))
}


func main(){
   navigateSiblings()
}

Output

$ go run example.go
Trending repositories on GitHub today · GitHub
byoungd / English-level-up-tips-for-Chinese
cjbarber / ToolsOfTheTrade
TheAlgorithms / Python

package main
import (
    "fmt"
    "log"
    "strings"
    //"io"
    //"os"
    "net/http"
    "github.com/PuerkitoBio/goquery"
)

func githubTrendingScraper(){
    resp, err := http.Get("https://github.com/trending")
    if err != nil{
        log.Fatal(err)
    }
    defer resp.Body.Close()
    if resp.StatusCode != 200{
        log.Fatalf("Status code error: %d %s", resp.StatusCode, resp.Status)
    }
    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil{
        log.Fatal(err)
    }

    fmt.Println(doc.Find("title").Text())
    doc.Find("ol li").Each(func (i int, s *goquery.Selection){
        repositoryName := strings.TrimSpace(s.Find("h3").Text())
        totalStarsToday := strings.TrimSpace(s.Find(".float-sm-right").Text())
        href, has_attr := s.Find("a").Attr("href")
        if !has_attr{
            href = "No valid url found"
        }
        fmt.Println(repositoryName, "\t", totalStarsToday, "\t", "https://github.com" + href)
    })

}


func main(){
   githubTrendingScraper()
}

Output

$ go run example.go
Trending repositories on GitHub today · GitHub
you-dont-need / You-Dont-Need-Momentjs      625 stars today https://github.com/you-dont-need/You-Dont-Need-Momentjs
ripienaar / free-for-dev                    476 stars today https://github.com/ripienaar/free-for-dev
Nozbe / WatermelonDB                        407 stars today https://github.com/Nozbe/WatermelonDB
cjbarber / ToolsOfTheTrade                  392 stars today https://github.com/cjbarber/ToolsOfTheTrade
byoungd / English-level-up-tips-for-Chinese 332 stars today https://github.com/byoungd/English-level-up-tips-for-Chinese
TheAlgorithms / Python                      316 stars today https://github.com/TheAlgorithms/Python
thedaviddias / Front-End-Checklist          304 stars today https://github.com/thedaviddias/Front-End-Checklist
zziz / pwc                                  274 stars today https://github.com/zziz/pwc
dawnlabs / carbon                           249 stars today https://github.com/dawnlabs/carbon
CyC2018 / CS-Notes                          201 stars today https://github.com/CyC2018/CS-Notes
Avik-Jain / 100-Days-Of-ML-Code             206 stars today https://github.com/Avik-Jain/100-Days-Of-ML-Code
donnemartin / system-design-primer          188 stars today https://github.com/donnemartin/system-design-primer
mariusandra / pigeon-maps                   192 stars today https://github.com/mariusandra/pigeon-maps
Snailclimb / JavaGuide                      165 stars today https://github.com/Snailclimb/JavaGuide
JavaNoober / BackgroundLibrary              154 stars today https://github.com/JavaNoober/BackgroundLibrary
crossoverJie / JCSprout                     141 stars today https://github.com/crossoverJie/JCSprout
Microsoft / nni                             153 stars today https://github.com/Microsoft/nni
PansonPanson / Java-Notes                   146 stars today https://github.com/PansonPanson/Java-Notes
date-fns / date-fns                         153 stars today https://github.com/date-fns/date-fns
sindresorhus / ky                           149 stars today https://github.com/sindresorhus/ky
mciastek / sal                              145 stars today https://github.com/mciastek/sal
rwv / chinese-dos-games                     134 stars today https://github.com/rwv/chinese-dos-games
vuejs / vue                                 124 stars today https://github.com/vuejs/vue
GoogleCloudPlatform / open-match            137 stars today https://github.com/GoogleCloudPlatform/open-match
lin-xin / vue-manage-system                 117 stars today https://github.com/lin-xin/vue-manage-system

Visualization with VisJS

2018-08-19T00:00:00+00:00

The intentions of this post is to host example code snippets so people can take ideas from it to make great visualization for themselves using visJS. VisJS is a dynamic, browser based visualization library. The library is designed to be easy to use, to handle large amounts of dynamic data, and to enable manipulation of and interaction with the data. The library consists of the components DataSet, Timeline, Network, Graph2d and Graph3d.

VisJS network [Nodes as images with label]

  </span>Bhishan's Services | TheTaraNights<span class="nt">

   href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis-network.min.css" rel="stylesheet" type="text/css" />
 rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
  
 onload="draw()">

 id="mynetwork">

VisJs Edges between nodes

  </span>Bhishan's Services | TheTaraNights<span class="nt">

   href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis-network.min.css" rel="stylesheet" type="text/css" />
 rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
  
 onload="draw()">

 id="mynetwork">

VisJS onclick() and onhover()

  </span>Bhishan's Services | TheTaraNights<span class="nt">

   href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis-network.min.css" rel="stylesheet" type="text/css" />
 rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
  
 onload="draw()">

 id="mynetwork">

Putting it all together [Bhishan’s Freelance Network]

  </span>Bhishan's Services | TheTaraNights<span class="nt">

   href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis-network.min.css" rel="stylesheet" type="text/css" />
 rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
  
 onload="draw()">

 id="mynetwork">

I post awesome content on programming tutorials and computer science in general. Subscribe to never miss an article and programming resources.

Web Scraping with NodeJS and Cheerio

2018-08-17T00:00:00+00:00

Web Scraping has been of an interest to a lot of businesses and individuals with the immense potential of the quantitative data available online. The data collected can entice the growth of an organization or a personal business. Through this post, we will see through examples on how NodeJS can be used to scrape content from a website. The intentions of this post is to host example code snippets so people can take ideas from it to build scrapers as per their needs using cheerio module in NodeJS. I will be using github’s trending page https://github.com/trending throughout this post for the examples, especially because it best suits for applying various cheerio methods.

Installation

npm install --save promise request-promise cheerio

Get html of a page:

var Promise = require("promise");
var request = require("request-promise");
var cheerio = require("cheerio");

function getHtml(){
    return new Promise(function(resolve, reject){
        request("https://github.com/trending")
            .then(cheerio.load)
            .then(console.log)
            .then(resolve)
            .then(reject)
    });
}

Using cheerio to get title from a page

function getHtml(){
    return new Promise(function(resolve, reject){
        request("https://github.com/trending")
            .then(cheerio.load, requestError)
            .then(scrapeContent)
            .then(resolve)
            .then(reject)
    });
}


function scrapeContent($){
    console.log($('title').text());
}

function requestError() {
    console.log("The trending page could not be loaded!");
    throw Error("Could not fetch html!");

}


getHtml();

Find single element by tag name, find multiple elements by tag name

function getHtml(){
    return new Promise(function(resolve, reject){
        request("https://github.com/trending")
            .then(cheerio.load, requestError)
            .then(scrapeContent)
            .then(resolve)
            .then(reject)
    });
}


function scrapeContent($){
    //console.log($);
    console.log($('title').text());
    var all_li_elements = $('ol').find('li');
    all_li_elements.each(function(item){
        console.log($(this).find('h3').text().trim());
    });
    
}

function requestError() {
console.log("The trending page could not be loaded!");
throw Error("Could not fetch html!");
}


getHtml();

Output

Trending repositories on GitHub today · GitHub
Microsoft / FASTER
MichaelMure / git-bug
google / python-fire
Droogans / unmaintainable-code
Avik-Jain / 100-Days-Of-ML-Code
pwxcoo / chinese-xinhua
JuliaLang / julia
r-spacex / SpaceX-API
IEEEKeralaSection / rescuekerala
react-tools / react-move
imhuay / Interview_Notes-Chinese
crossoverJie / JCSprout
benhoyt / goawk
salesforce / TransmogrifAI
Jeffail / benthos
aykevl / tinygo
astorfi / Deep-Learning-World
vuejs / vue
firehol / netdata
loveRandy / vue-cli3.0-vueadmin
trekhleb / javascript-algorithms
palmerhq / react-async-elements
messeb / ios-project-env-setup
jianstm / Schedule
kholia / OSX-KVM

Getting Attributes of an element

function getHtml(){
    return new Promise(function(resolve, reject){
        request("https://github.com/trending")
            .then(cheerio.load, requestError)
            .then(scrapeContent)
            .then(resolve)
            .then(reject)
    });
}


function scrapeContent($){
    //console.log($);
    console.log($('title').text());
    var all_li_elements = $('ol').find('li');
    all_li_elements.each(function(item){
        console.log("https://github.com/" + $(this).find('a').attr('href'));
    });
}

function requestError() {
    console.log("The trending page could not be loaded!");
    throw Error("Could not fetch html!");

}


getHtml();

Output

Trending repositories on GitHub today · GitHub
https://github.com//Microsoft/FASTER
https://github.com//MichaelMure/git-bug
https://github.com//google/python-fire
https://github.com//Droogans/unmaintainable-code
https://github.com//Avik-Jain/100-Days-Of-ML-Code
https://github.com//pwxcoo/chinese-xinhua
https://github.com//JuliaLang/julia
https://github.com//r-spacex/SpaceX-API
https://github.com//IEEEKeralaSection/rescuekerala
https://github.com//react-tools/react-move
https://github.com//imhuay/Interview_Notes-Chinese
https://github.com//crossoverJie/JCSprout
https://github.com//benhoyt/goawk
https://github.com//salesforce/TransmogrifAI
https://github.com//Jeffail/benthos
https://github.com//aykevl/tinygo
https://github.com//astorfi/Deep-Learning-World
https://github.com//vuejs/vue
https://github.com//firehol/netdata
https://github.com//loveRandy/vue-cli3.0-vueadmin
https://github.com//trekhleb/javascript-algorithms
https://github.com//palmerhq/react-async-elements
https://github.com//messeb/ios-project-env-setup
https://github.com//jianstm/Schedule
https://github.com//kholia/OSX-KVM

Using class name or other attributes to get element

function getHtml(){
    return new Promise(function(resolve, reject){
        request("https://github.com/trending")
            .then(cheerio.load, requestError)
            .then(scrapeContent)
            .then(resolve)
            .then(reject)
    });
}


function scrapeContent($){
    //console.log($);
    console.log($('title').text());
    var all_li_elements = $('ol').find('li');
    all_li_elements.each(function(item){
        console.log($(this).find('.float-sm-right').text().trim());
    });
}

function requestError() {
    console.log("The trending page could not be loaded!");
    throw Error("Could not fetch html!");

}


getHtml();

Output

Trending  repositories on GitHub today · GitHubAsset 1Asset 1
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today
stars today

Navigate childrens from an element

function getHtml(){
    return new Promise(function(resolve, reject){
        request("https://github.com/trending")
            .then(cheerio.load, requestError)
            .then(scrapeContent)
            .then(resolve)
            .then(reject)
    });
}


function scrapeContent($){
    //console.log($);
    console.log($('title').text());
    var all_li_elements = $('ol').children(); // using children()
    all_li_elements.each(function(item){
        console.log($(this).find('h3').text().trim());
    });
}

function requestError() {
    console.log("The trending page could not be loaded!");
    throw Error("Could not fetch html!");

}


getHtml();

Output

Trending repositories on GitHub today · GitHub
Microsoft / FASTER
MichaelMure / git-bug
google / python-fire
Droogans / unmaintainable-code
Avik-Jain / 100-Days-Of-ML-Code
pwxcoo / chinese-xinhua
JuliaLang / julia
r-spacex / SpaceX-API
IEEEKeralaSection / rescuekerala
react-tools / react-move
imhuay / Interview_Notes-Chinese
crossoverJie / JCSprout
benhoyt / goawk
salesforce / TransmogrifAI
Jeffail / benthos
aykevl / tinygo
astorfi / Deep-Learning-World
vuejs / vue
firehol / netdata
loveRandy / vue-cli3.0-vueadmin
trekhleb / javascript-algorithms
palmerhq / react-async-elements
messeb / ios-project-env-setup
jianstm / Schedule
kholia / OSX-KVM

The .children will only return the immediate childrens of the parent element.

Navigating previous and next siblings of an element

function getHtml(){
    return new Promise(function(resolve, reject){
        request("https://github.com/trending")
            .then(cheerio.load, requestError)
            .then(scrapeContent)
            .then(resolve)
            .then(reject)
    });
}


function scrapeContent($){
    //console.log($);
    console.log($('title').text());
    var fifth_li_element = $('ol').find('li').eq(4); // use eq(index) to access via indices.
    var fourth_li_element = fifth_li_element.prev(); // previous sibling
    console.log(fourth_li_element.find('h3').text().trim());
    var sixth_li_element = fifth_li_element.next(); // next sibling
    console.log(sixth_li_element.find('h3').text().trim());
}

function requestError() {
    console.log("The trending page could not be loaded!");
    throw Error("Could not fetch html!");

}


getHtml();

function getHtml(){
    return new Promise(function(resolve, reject){
        request("https://github.com/trending")
            .then(cheerio.load, requestError)
            .then(scrapeContent)
            .then(resolve)
            .then(reject)
    });
}


function scrapeContent($){
    //console.log($);
    var all_li_elements = $('ol').find('li');
    all_li_elements.each(function(item){
        repository_name = $(this).find('h3').text().trim();
        total_stars_today = $(this).find('.float-sm-right').text().trim();
        repository_url = 'https://github.com/' + $(this).find('a').attr('href');
        console.log(repository_name, "\t", total_stars_today, "\t", repository_url);
    });
}

function requestError() {
    console.log("The trending page could not be loaded!");
    throw Error("Could not fetch html!");

}

getHtml();

Output

Microsoft / FASTER                946 stars today      https://github.com//Microsoft/FASTER
google / python-fire              647 stars today      https://github.com//google/python-fire
MichaelMure / git-bug             578 stars today      https://github.com//MichaelMure/git-bug
Droogans / unmaintainable-code    228 stars today      https://github.com//Droogans/unmaintainable-code
pwxcoo / chinese-xinhua           198 stars today      https://github.com//pwxcoo/chinese-xinhua
Avik-Jain / 100-Days-Of-ML-Code   188 stars today      https://github.com//Avik-Jain/100-Days-Of-ML-Code
JuliaLang / julia                 170 stars today      https://github.com//JuliaLang/julia
aykevl / tinygo                   169 stars today      https://github.com//aykevl/tinygo
IEEEKeralaSection / rescuekerala  96 stars today       https://github.com//IEEEKeralaSection/rescuekerala
benhoyt / goawk                   152 stars today      https://github.com//benhoyt/goawk
r-spacex / SpaceX-API             150 stars today      https://github.com//r-spacex/SpaceX-API
crossoverJie / JCSprout           129 stars today      https://github.com//crossoverJie/JCSprout
imhuay / Interview_Notes-Chinese  121 stars today      https://github.com//imhuay/Interview_Notes-Chinese
trekhleb / javascript-algorithms  113 stars today      https://github.com//trekhleb/javascript-algorithms
salesforce / TransmogrifAI        107 stars today      https://github.com//salesforce/TransmogrifAI
loveRandy / vue-cli3.0-vueadmin   110 stars today      https://github.com//loveRandy/vue-cli3.0-vueadmin
astorfi / Deep-Learning-World     102 stars today      https://github.com//astorfi/Deep-Learning-World
vuejs / vue                       97 stars today       https://github.com//vuejs/vue
Jeffail / benthos                 104 stars today      https://github.com//Jeffail/benthos
palmerhq / react-async-elements   92 stars today       https://github.com//palmerhq/react-async-elements
firehol / netdata                 89 stars today       https://github.com//firehol/netdata
jesseduffield / lazygit           84 stars today       https://github.com//jesseduffield/lazygit
jianstm / Schedule                75 stars today       https://github.com//jianstm/Schedule
shadowsocks/shadowsocks-windows   67 stars today       https://github.com//shadowsocks/shadowsocks-windows
mawww / kakoune                   71 stars today       https://github.com//mawww/kakoune

There is another version of this article using same set of examples using BeautifulSoup in Python. You should read it too. https://www.thetaranights.com/web-scraping-beautifulsoup-python/

Google APIs and Python – Part II

2018-08-16T00:00:00+00:00

Google services are cool and you can build products and services around it. We will see through examples how you can use various google services such as spreadsheet, slides and drive through Python. I hope people can take ideas from the following example to do amazing stuffs with Google services. There is a part one to this article where I walked through procedure to enable Google APIs, installation of required packages in Python, authentication and demonstrated individual examples of Sheets, Drive and Slides API. https://www.thetaranights.com/brief-introduction-to-google-apissheets-slides-drive/ . In this article however, we will integrate Sheets, Drive and Slides API altogether.

The Idea

We will use data from a sheet which contains some statistics about a few applications/websites. The end goal is to create a presentation slide, add a background image to it, add content from the sheet to the slide and also some other cool stuffs. All the resources used in the following examples are public so you can follow along.

I will be using following resources throughout the example

Presentation Template https://docs.google.com/presentation/d/1wLimfuGw1pqZZJvkc15lOfD7LSEAWCjuIlhbrOaiulE/edit?usp=sharing Background Images https://drive.google.com/file/d/0B54qUrMD2GDIa2syZWF3OE5xSUk/view?usp=sharing https://drive.google.com/file/d/1riNC6PykmjmokXa1My3KyseoKO8MrZGI/view?usp=sharing https://drive.google.com/file/d/10RUMiifQ7XQn88kSaDy2G83U5sjTeyBS/view?usp=sharing https://drive.google.com/file/d/1hc8UOmxbFzAS7IaASRZwOftnBBRfBjIs/view?usp=sharing Spreadsheet https://docs.google.com/spreadsheets/d/1xpjQkF692lNnTsfOckVll2OPTa659ZCuK3JezDSkris/edit?usp=sharing

Create Presentation Slides from Sheets data and Drive images using Python

from googleapiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

TEMPLATE_FILE = "TEM_F"

SCOPES = ('https://www.googleapis.com/auth/spreadsheets','https://www.googleapis.com/auth/drive')

CLIENT_SECRET = 'client_secret_760822340075-i0ark1h51pnbhii5dgafug3k4g1nodb8.apps.googleusercontent.com.json' # download from google console after activating apis

store = file.Storage('storage.json') # doesn't matter if not present, you will be prompted to accept access to google resources on your account and a token will be generated that is stored inside storage.json with requested previliges.

credz = store.get()

if not credz or credz.invalid:
    flow = client.flow_from_clientsecrets(CLIENT_SECRET, SCOPES)
    credz = tools.run_flow(flow, store)

HTTP = credz.authorize(Http())

SHEETS = discovery.build('sheets', 'v4', http=HTTP)

SLIDES = discovery.build('slides', 'v1', http=HTTP)

DRIVE = discovery.build('drive', 'v3', http=HTTP)


presentation_template_file_id = "1wLimfuGw1pqZZJvkc15lOfD7LSEAWCjuIlhbrOaiulE" # the template has been made public.
# name of the presentation file
DATA = {'name':'MobileApplicationsReport'}

PRESENTATION_ID = DRIVE.files().copy(body=DATA, fileId="1wLimfuGw1pqZZJvkc15lOfD7LSEAWCjuIlhbrOaiulE").execute()['id']
print(PRESENTATION_ID)

sheet_ID = '1xpjQkF692lNnTsfOckVll2OPTa659ZCuK3JezDSkris' # the sheet where we fetch data from to populate to the slides.

application_statistics = SHEETS.spreadsheets().values().get(range='Sheet1', spreadsheetId=sheet_ID).execute().get('values') # all the data from the sheet as lists including headers.

print(application_statistics)

presentation_details = SLIDES.presentations().get(presentationId=PRESENTATION_ID).execute()

slides_data = presentation_details.get('slides', [])[0]

page_id = slides_data['objectId'] # page id of the first slide of the presentation.

for each_data in application_statistics[1:]: # skip the headers.
    # duplicate slide for the next cycle before replacing content on a slide since we are using method of replacing text from the slide to populate data.
    reqs = [{"duplicateObject": {"objectId": page_id}}]
    copy_slide_rsp = SLIDES.presentations().batchUpdate(body={'requests':reqs}, presentationId=PRESENTATION_ID).execute()
    
    IMG_ID = each_data[10] # the id of the image present on google drive which we intend to have as a background image to this particular slide.
    img_url = '%s&access;_token=%s' % (DRIVE.files().get_media(fileId=IMG_ID).uri, credz.access_token)
    print("Image url", img_url)

    # prepare a bulk requests that basically replaces the text from the template with the actual data from the sheets.
    bulk_requests = [
        {'updatePageProperties':{'objectId':page_id, 'pageProperties':{'pageBackgroundFill':{'stretchedPictureFill':{'contentUrl':img_url}}}, 'fields':'pageBackgroundFill'}},
        {'replaceAllText':{'containsText':{'text':'', 'matchCase':True}, 'replaceText':each_data[1], "pageObjectIds": [page_id]}},
        {'replaceAllText':{'containsText':{'text':'', 'matchCase':True}, 'replaceText':each_data[2], "pageObjectIds": [page_id]}},
        {'replaceAllText':{'containsText':{'text':'', 'matchCase':True}, 'replaceText':each_data[3], "pageObjectIds": [page_id]}},
        {'replaceAllText':{'containsText':{'text':'', 'matchCase':True}, 'replaceText':each_data[8], "pageObjectIds": [page_id]}},
        {'replaceAllText':{'containsText':{'text':'', 'matchCase':True}, 'replaceText':each_data[7], "pageObjectIds": [page_id]}},
        {'replaceAllText':{'containsText':{'text':'', 'matchCase':True}, 'replaceText':each_data[6], "pageObjectIds": [page_id]}},
        {'replaceAllText':{'containsText':{'text':'', 'matchCase':True}, 'replaceText':each_data[9], "pageObjectIds": [page_id]}}
    ]
    bulk_update_response = SLIDES.presentations().batchUpdate(body={'requests':bulk_requests}, presentationId=PRESENTATION_ID, fields='').execute().get('replies')

    page_id = copy_slide_rsp['replies'][0]['duplicateObject']['objectId'] # update the page id as the one that was duplicated so we now can work on this slide.

delete_final_page = SLIDES.presentations().batchUpdate(body={'requests':[{"deleteObject": {"objectId": page_id}}]}, presentationId=PRESENTATION_ID, fields='').execute().get('replies')

Output Presentation Created:

After successful running of the above program, following presentation was generated. https://docs.google.com/presentation/d/1h9YqUnCWu5pxXmW3rs_9rKmMsVeJM9I8nIBGbr25pME/edit?usp=sharing

Bhishan Bhandari / Product and Engineering

Python and Excel

Installing openpyxl

Open an existing document in python

Selecting sheets from the workbook/document

When the cell being accessed doesn’t have a value

Accessing cell values using row, column directive

Reading values from excel sheet

Creating excel document

Idiomatic Python - Use of Falsy and Truthy Concepts

An example of a code snippet that is considered bad:

An example of code snippet that is considered relatively wise:

Basic File Operations - Golang

Creating an empty file

Reading a File

Writing to a file

Truncate File

File Information

Check if file exists or not

Move/Rename File

Copy File

Going serverless with Chalice and AWS lambda

Prerequisites for chalice[AWS Credentials]

Installing Chalice

Creating simple API with chalice

Auto-generated app.py

Running the API locally

Going serverless with chalice through AWS lambda

Top 10 Most Subscribed Youtube Channel Visualization[2013-2018]

Python, Boto and AWS EC2

Filter instances based on tag names from the AWS inventory

Boto2

Boto3

Boot/Shutdown an instance/instances from the AWS inventory

Boto2

Boto3

Create Instances based on various metrics

Boto2

Boto3

Send remote commands to an EC2 instance

Web Scraping with Golang and Goquery

Installation

Get html of a page

Using goquery(golang library) to get title from a page

Using goquery, Find single element by tag name, find multiple elements by tag name

Getting Attributes of an element

Using class name or other attributes to get element

Navigate childrens from an element

Navigating previous and next siblings of an element

Putting it all together(Github Trending Scraper using Golang)

Visualization with VisJS

VisJS network [Nodes as images with label]

VisJs Edges between nodes

VisJS onclick() and onhover()

Putting it all together [Bhishan’s Freelance Network]

Web Scraping with NodeJS and Cheerio

Installation

Get html of a page:

Using cheerio to get title from a page

Find single element by tag name, find multiple elements by tag name

Getting Attributes of an element

Using class name or other attributes to get element

Navigate childrens from an element

Navigating previous and next siblings of an element

Putting it all together(Github Trending Scraper)

Google APIs and Python – Part II

The Idea

Create Presentation Slides from Sheets data and Drive images using Python

Output Presentation Created: