Skip to main content


Day 24 Big Data on Hadoop Udemy | Hive

We got our image started in the virtual sand box. We needed to disable hyper-v as Windows 10 doesn't allow Virtual Box Hyper-V and Hyper-V on the windows 10. We downloaded the dataset , now uploading the dataset in Hive Ambari Sandbox allows us to directly use Hive from the  url. we can import data in the csv format and query data as sql relational although it is not stored relational. we can also visualize the selected data columns. Difference between RANK , DENSE RANK and ROW_NUMBER ROW_NUMBER will always give unique RANK will give same rank for the first 4 same but 5th rank for the 5th . DENSE_RANK will give first 4 same  but 2 for the 5th like a group by. Scrolling Page Title http://www.techt
Recent posts

Day 23 Big Data on Hadoop Udemy | Arduino

So my wife has secured a job in Luxembourg and I need to board along. I looked at how I could maximise my chances. Machine Learning has really good opportunities but I need time probably atleast 6 months. 3 things always crept up my mind Data Science, AI/ML and Big Data. I foound Big Data to be interesting and more achievable in 3 months target as I'm familiar with most technologies and needed to upgrade myself. I had a choice of choosing between Microsoft Program in Big Data vs Udemy Big Data. I chose Udemy as it's slightly lesser on schedule and more precise on content, also the pricing is very attractive, which should have been a similar case for Microsoft but seems like 1000$ for a certificate which would teach similar skills just on Microsoft stack seemed little steep. So I've bought the Udemy - Ultimate Hadns on Big Data. Another exciting thing for me was the trainer is an Amazon and IMDB ex employee and I actually quite like their UX and the way one could sear

Day 22 Microsoft AI Math | Arduino

Linear Equations y=mx+b m slope and b coefficient System of Equations x+y =10 2x+5y=34 one of the easiest way is to find intersection point, if you substitute x n y as 0 and plot you will get the intersection. another way is to add the 2 equations both eliminating x or y and getting the point of intersection.

Day 21 Microsoft AI Math | Arduino

So, instead of continuing in python programming course , I jumped on to Math for ML as that is more essential in terms of getting to it. I have been programming a while in python so it seems to be not an issue , although I need to keep up with all packages but since I keep doing the demo in the course hand typing myself , I start getting familiarized, this is a good point for lot of developers , instead of copy pasting or simp[ly uploading the Ipython notebook , start typing literally almost everything , the learning would be higher plus higher retaining, although the former will allow to finish faster but one may forget as he never encountered any issue while doing those , in my case I encounter so many setup issues that it helps retaining lot of things naturally. So in the same light and learning spirit , I want to dive into maths and understand something , quite eager. Distributive Property

Day 21 Microsoft AI Python | Arduino

So now I have begun my new course Python in the 10 series. DataFrame Iterating Note  − Do not try to modify any object while iterating. Iterating is meant for reading and the iterator returns a copy of the original object (a view), thus the changes will not reflect on the original object. XML Etree Element Tree One thing I realized how .Net had made the xml a piece of cake. Especially after Linq to Xml , we took most of things for granted , never did a xml fail a parse. In python I had to remove xml namespaces and then parse to get things working def removexmlnamespaces(xst):         it = ET.iterparse(StringIO(xst))     for index, el in it:         if '}' in el.tag:             el.tag = el.tag.split(

Day 20 - Microsoft Artifical Intelligence | Arduino Robotic Kit

Data Science Virtual Machine Wow , seems a neat concept Seems I cannot change HDD to SDD once I create the VM Cognitive ToolKit - Examples of Deep Learning in python I have just copy pasted below content from the App Developer Resources Cognitive Services For an overview of the Cognitive Services APIs, take a look at  Getting Started with Microsoft Cognitive Services  on Channel 9. Then you can dive into the documentation for individual Cognitive Services APIs, which often includes tutorials to help you get started (note that this is not an exhaustive list!): Linguistic Analysis Text Analytics Bing Speech Translator Language Understanding Intelligent Services (LUIS) Custom Vision Computer Vision Face Video Indexer Building Bots For in-depth trai

Day 19 - Microsoft Artifical Intelligence | Arduino Robotic Kit

Finally I think Visual Studio has a fix for Anconda and Python envronments. It seems to recognise anaconda default environment and create more, also now I can see both packages pip and conda separately. It seems to make more sense as it lesser loaded . Installing av in python Git Clone Video Indexer API List Videos Find Video Basic Info by Video Id Upload Video Thumbanial View IPython Magic VIdoe Indexer API allows to view thumbnail of key images you can embed video edit from video ai inline Moving to BOTS Need to deal with NodeJs as I have downloaded the bOTS API source code for NodeJs environment. gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable. Bot Framework emulator You can use the bot to embed as iframe in

Day 18 - Microsoft Artifical Intelligence | Arduino Robotic Kit

Image Classifier Classify an image whether the tag matches I couldn't get it to work exactly. Azure Computer Vision API very interesting API , probably the most that I like so far of all, gives features about any picture and seemed fairly accurate. Most of the cognitive APIs body accepts json string instead of json data Face Detection APIs Can give certain attributes, match 2 faces Jupyter is formerly known as IPython so both are closely interchangeable None of the interactive windows seem to work Failed to start the kernel XSRF cookie does not match POST argument OpenCV Python Library Finally got something working for certain packages insta

Day 17 - Microsoft Artifical Intelligence | Arduino Robotic Kit

Uninstall packages What is difference between cumulative distribution and normal distribution in histogram Gaussian Filter a 3 by 3 matrix is taken where in the center value is formed by average of beside values, this technique is moved across the whole image to reduce noise by averaging out all pixels sigma is the mask size Median Filter Edge Detection Algorithm Sobel Edge Detection Algorithm similar to filters , workd with a mask to find a gradient for x axis and y axis Harris Corner Algorithm Didn't work for median filtered image Custom VIsion AI Project Selecting images and tagging them , train them

Day 16 - Microsoft Artifical Intelligence | Arduino Robotic Kit

For anaconda packages , use Anaconda navigator which identifies correctly the missing dependencies. IPython doesn't work correctly need to move to jupyter_client Profiling in VIS for python Installing Package in Notebook a-pip-package-from-within-a-jupyter-notebook-not-working Speech Translator Services and Language Understanding Intelligence Services LUIS Web Chat BOT https://docs.microsof

Day 15 - Microsoft Artifical Intelligence | Arduino Robotic Kit

Azure ECOSystem Azure Application Insights Just add a tag to client side to Feature Hashing Giving a word or phrase an integer value indicating the relevance of that text in a given text or dataset. Created 32k column names , hence takes time to open the next dialog box Cannot apply SQL Transformation Need to use Select Columns Filter for publishing web service Sentimental Analysis Excel online Icon Sets- something new in excel as well Azure Cognitive Services for Text Analytics Error in urllib.request or conn.request

Day 14 - Microsoft Artifical Intelligence | Arduino Robotic Kit

Accessing y index Ternary Operator Python Dynamic Programming Language Focussed on Readibility Interpreted Language as vs Compiled Language Multiparadigm : Structure , object and Funtional oriented NLTK Download and punkt What are these pickle files C:\Users\rameezahmed.sayad\AppData\Roaming\nltk_data\tokenizers\punkt\PY3 Pickling and Unplicking ,pickle files Pandas Dataframe Document Word Frequency and Inverse document Frequency Term Frequency or Word Frequency - is the frequency of a word in a document against tot

Day 13 - Microsoft artificial Intelligence | Arduino Robotic Kit

Using Visual Studio for python -- back to debugging mode , feels amazing. So finally after managing to get 4 18650 batteries , 2 of them are discharged , now not sure how to recharge them. Good Resources for AI Python Math Operators Reading an Input form cmd line with python Convert string to int in Python

Day 12 - Microsoft artificial Intelligence | Arduino Robotic Kit

Learning about electricity fundamentals. So we need 7v to 12v to power in battery mode. We got a 9v battery just to see if it works, but seems like the current might not be enough. So it's final seems , the current provided by the 9v battery is quite less and needs more amps. 18650 Battery Now back to Artificial Intelligence USING CURL to DOWNLOAD files For opening Notebook https://azureml

Day 11 Microsoft artificial Intelligence | Arduino Robotic Kit

Running motor using USB Voila , Motor, LED , Ultrasonic and Servo in action Only 1 motor can run from USB , both the motors cannot run simultaneously from USB voltage. 2 motors can be run or the 5v pins can be run at once.

Day 10 - Microsoft Artificial Intelligence | Arduino Robotic Kit

Excel Online Try Except Python Azure Notebooks Urllib2 to urllib Power Arduino using AC power Vin pin can also be used to power Arduino Servo Motor is for rotating using Arduino Arduino Shield is used to power and control motor. It has default pins which are used for sensing current and controlling the speed and direction of the motor.

Day 9 - Microsoft Artificial Intelligence | Arduino Robotic Kit

Motor Shield for Arduino motor shield is required for Bluetooth , GPS , GSM and Servo Motor in order to generate a stable and controlled current.  Connecting directly to Arduino board , the equipments may start drawing large current damaging the board. First Experiment with Arduino - LED Test. The RGB LED sensor has 4 points , 1 positive for 5 V connection and 1 each for red , green , blue . Pass accordingly LOW and HIGH to arduino Red , Green and Blue.

Day 8 - Microsoft AI | Arduino Robotic Kit

Arduino Robotic Kit Just got my Arduino robotic kit , quite excited :-) , Let us start exploring the various components. HC SR 04 Ultrasonic Ranging Module HC - SR04 provides 2cm - 400cm non-contact measurement function, the ranging accuracy can reach to 3mm Vcc , Trigger , Echo n Ground  You only need to supply a short 10uS pulse to the trigger input to start the ranging, and then the module will send out an 8 cycle burst of ultrasound at 40 kHz and raise its echo. The Echo is a distance object that is pulse width and the range in proportion

Day 7 - Microsoft Professional Program for Artificial Intelligence track

2 class logistic regression used to predict only 2 outcomes For  Label column , click  Launch column selector , and choose a single column that contains outcomes the model can use for training. For classification problems, the label column must contain either  categorical  values or  discrete  values. Some examples might be a yes/no rating, a disease classification code or name, or an income group. If you pick a noncategorical column, the module will return an error during training. For regression problems, the label column must contain  numeric  data that represents the response variable. Ideally the numeric data represents a continuous scale for creating web service , all steps need to be run

Day 6 - Microsoft Professional Program for Artificial Intelligence track

Scatter Plot Ready Module in AZMLS For removing an outlier , we can use sql transformation to update any row or values Why use matplotlib agg Setting utlier to threshhold , By using Clip Values , upper na dlower threshold can be set. Convert to Indicator Values to add a column IsCategoricalValue 0 or 1 based on the no of values in the selected feature Supervised Learning Nearest neighbour, Na├»ve Bayes, Decision Trees, Regression  UnSupervised Learning K- means Clustering Algorithm ReInforcement Learning The machine/ software agent trains itself on a continual basis based on the environment it is exposed to, and applies it’s enriched knowledge to solve business problems.  Markov Decision Process “A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on

Day 5 - Microsoft Professional Program for Artificial Intelligence track

PCA vs Mean - If probabilistic is used against a single column , it is equivalent to mean as it will used the single column. MICE - multiple imputation chained equations Normal Distribution - Bell Curve Seaborn Indentation in Python Anaconda does not have Seaborn yet Normalizing Data - there are 5 technieques , even log , mainly because if there are large values in the data set which do not dominate the prediction Z SCORE score using std  deviation MIN MAX score between 0 and 1

Day 4 - Microsoft Professional Program for Artificial Intelligence track

Decision Tree Algorithm Assigning a label to the problem instance, Eg classifying gender so male or female become labels. Single Decision Tree single Decision Tree Ensemble Decision Tree multiple nodes Homogenous Sets- similar data or similar structure Heterogeneous Sets - mixed or unsimilar data Finding homogeneity - Information Gain Gini Impurity Beautiful Soup Python package for parsing HTML. Pandas Python package for reading HTML to dataframe , data analysis package. Panda Python package - categorical function for assigning integer to alphanumeric data. sklearn decision tree classifier Building own ML - Visual through ML Studio . Code first through Python notebooks. Consuming built in ML models - Azure cognitive services and bots

Day 3 - Microsoft Professional Program for Artificial Intelligence track

AI Cheat Sheet Jargons for ML Training Data - after raw data is processed Columns in Training Data is called Features Regression - Supervised Learning - line that best fits , least error Classification - Supervised Learning classify into classes Clustering - Unsupervised Learning create clusters Decision Tree Neural Network Bayesian Algorithm K-means Clustering Data Scientists can choose the features more appropriately who have the domain knowledge. Feature Engineering - includes cleaning , normalizing , rebalancing , grouping , filtering  raw data

Day 2 - Microsoft Professional Program for Artificial Intelligence track

Back to basics Learning Binary Search Tree and algorithms for Binary Tree Big  O notation O(1) - exactly 1 computation to do the operation O(n) - computation time is equal to the size of array O(n square) - computation to nested for-each O(log (n)) - divide and conquer algorithms O(n log(n)) - Insertion Sort - iterate each element , check if it is lower than any of the element at the left and insert in between and move other elements greater than it. O(n square) , best O(n) - Bubble Sort - keep comparing side by side each element and swap O(n square) , best O(n) - if no swap at the first iteration Selection Sort - assume the first is smallest and find the smallest at first position, increment each position and check if anything is smaller. O(n square) , best O(n2) Heap Sort - elements are stored in binary tree. root node will be greater than child node.swap root and child , finally swap the last and first node , t

Day 1 - Microsoft Professional Program for Artificial Intelligence track

Based on my friend's request , I have begun to write more on it. So it's been a long time since I have left Data Science in some lurk , but honestly I did struggle a lot to cope up with it. Plus because of timeframes , I scrambled through Data Science initial models , but I couldn't gather a lot except R programming and concepts of Data Science , ML and AI. Data Science seems a tough hill for me , by tough hill I mean way more time and effort , I got busy in work and then somehow , it's been pending , Suprisingly I didn't come across Microsoft Certification Program because that should be the defacto choice for a .Net Developer having spent more than 8 years professionally and 19 years with Microsoft and it's ecosystem. So I began seeing Microsoft Artifical Intelligence course it seems very exciting , although I remember my stint from GCP (Google Cloud Platform ) , it was a big thing doing some things I did very naturally with SQL or Excel. https://www.

Day 1 - Azure Learning - Docker

Docker Containers - switching "docker pull" fails in windows 10 [Windows] Cannot connect to plugin server -- wsarecv: An existing connection was forcibly closed by the remote host. Disable Docker Support Remove Docker from Visual Studio project Turn Off Docker Support

Common Errors - NPM , Nodemon

Can't seem to get Node js and NPM up and running: Reference Error on every command 'npm' is not recognized as internal or external command, operable program or batch file Node / Express: EADDRINUSE, Address already in use - Kill server

Visual Studio Node JS - Initial Setup

Let's keep it precise and small steps. 1. You need Visual Studio 2017 Community or Professional. 2. Select NodeJs development package. 3. You need NodeJs runtime to be installed from 4. Start a new project from VS 2017 , TypeScript->NodeJs 5. You might get an error , http or project not found. 6. Check the reference of the TypeScript project. 7. If you see the warning icon indicating that mode reference missing. 8. Right click and click Install from npm. 9. I you receive an error message , just follow the instructions. 10. Specify the node.exe path in the proj properties.C:\node-v8.9.4-win-x64\node.exe 11. Build and voila , let's start.

Day 25 R Programming

Day 24 (quite a break) Back to Git Issues

So after 3 weeks of sis in law's marital rituals and back to office loaded with tons of work , I finally get some time to get back. And only to realise that it's been more than a month now , I'm already beginning to forget . First day and git issues start , Apparently before leaving I had pulled the working code and downloaded rstudio and r so that I could probably learn something in my vacation (high hopes) , didnt touch a thing. Now when i come back I realize that there were certain changes I had done on this machine and from RStudio I had even commited but it doesn't show up in my github repo or the other machine when I pull. Something was wrong. Figured out it was because of the origins , I didn't add .git to the end of the Repo while adding remote origin and that indeed caused some issues , surprisingly RStudio didn't error out and even when I check it's history it shows me , So finally recreated the origin , pull and pushed the code thankfully so

Day 23 Git Crashes

** Please tell me who you are. Run   git config --global ""   git config --global "Your Name" to set your account's default identity. Omit --global to set the identity only in this repository. fatal: empty ident name (for <(NULL)>) not allowed Git - fatal: Unable to create '/path/my_project/.git/index.lock': File exists.

Day 23 R Objective Functions. Plotting , Date Time

Objective functions could be imagined as essence of a constructor in a function . this gives the ability to reuse or declare the parameters to the function and keep it instantiated instead of passing the parameters each time. Learned a little bit of plotting and the different types of plot , probably the easiest of the visualizations. Mode, Class and Type of R objects t2, like all POSIXlt objects, is just a list of values that make up the date and | time. Use str(unclass(t2)) to have a more compact view. >  > str(unclass(t2)) List of 11  $ sec   : num 18.5  $ min   : int 32  $ hour  : int 16  $ mday  : int 23  $ mon   : int 3  $ year  : int 117  $ wday  : int 0  $ yday  : int 112  $ isdst : int 0  $ zone  : chr "AST"  $ gmtoff: int 10800  - attr(*, "tzone&q

Day 22 R Coursera

Day 21 R Continuing in Coursera

So I'm just done with GCP on Coursera , a brief introduction to the set of tools provided by the Google Cloud Platform and practical hands on lab on certain things to give the realization how easy it is to get things started up. Partial matching of function argument The three-dots construct in R

Day 20 Google APIs, Google Application Default Credentials

Searching for objects attribute value, it has to be Datastore . Remember that BigTable, you can only search by key.  High-throughput writes of wide-column data. Well, that is BigTable , right, because it's supporting high-throughput writes.  Warehousing structured data. So what's the data warehouse technology on Google Cloud? That's, which one, BigQuery .  To create and test new machine learning methods. Well, if you're writing new machine learning methods, then TensorFlow .  Develop Big Data algorithms interactively in Python.Well, interactive development in Python is done best with Datalab .   Well, interactive No-ops, custom machine learning applications at scale. No-ops ML at scale, then that's a role for Cloud ML.  Automatically reject inappropriate image content. Rejecting image content where it is inappropriate. Well, that could be the Vision API. So you could use a Vision API to basically see if this is safe content or not safe content. 

Day 19 TensorFlow,2&seed=0.98991&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false Feature Engineering For Machine Learning in TensorFlow, for training the model , the data has to be numeric ( also not categorical variable - not holdeing weightage for that value but just a representation or classification like day of the

Day 18 GCP DataLab, Big Query from Client Side , Pandas- Python

'sudo su -' vs 'sudo -i' vs 'sudo /bin/bash' - when does it matter which is used, or does it matter at all? docker ps  will show only running containers by default. To see all containers:  docker ps -a root1234 - paraphrase DataLab gives the ability to share a notebook with other people , at the same time use the cloud for computing n storage. Python Data Analysis Library

Day 17 Periodic Data Science

Becoming a Data Scientist:  Profiling Cisco’s Data Science  Certification Program Wow , found this on R-Bloggers , quite awesome , I have certain blocks from each legend but still a logn way to go.

Day 16 DataProc , Goole Cloud Solutions

The Beginner’s Guide to Nano, the Linux Command-Line Text Editor So far from the course Google Cloud Platform , I got the main advantage using GCP is invariant of which tehcnology whether they are Sql Db or NoSql DB , Spark or Hadoop , GCP offers to run your pieces of programs to be run for a specific amount of time unlike in other scenario where one would end up many hardware or software in order to perform these heavy processing tasks. Microservices Architecture on Google App Engine

Day 15 GCP CloudSql Lab, Hadoop

bash ./ cd training-data-analyst/CPB100/lab3a Note : If you lose your Cloud Shell VM due to inactivity, you will have to reauthorize your new Cloud Shell VM with Cloud SQL. For your convenience, lab3a includes a script called  that you can run. You have to make sure the console IP is specified in the Autorized Networks PySpark Cheat SHeet DataProc is google managed Hadoop Spark Pig Hive Program The Hadoop Ecosystem Table HBase, Sqoop, Flume and More: Apache Hadoop Defined,_Sqoop,_Flume_and_More:_Apache_Hadoop_Defined http://www

Day 15 GCP Recommendations , Cloud SQL PySpark DataProc

Collaborative Filtering - RDD-based API PySpark Managed Hadoop & Spark Fully-Managed PostgreSQL  BETA  & MySQL Cloud sql can run 1 petabit per second

Day 14 Swirl

install.packages("swirl") | You can exit swirl and return to the R prompt (>) at any time by pressing the Esc key. If you are | already at the prompt, type bye() to exit and save your progress. When you exit properly, you'll see a | short message letting you know you've done so. | When you are at the R prompt (>): | -- Typing skip() allows you to skip the current question. | -- Typing play() lets you experiment with R on your own; swirl will ignore what you do... | -- UNTIL you type nxt() which will regain swirl's attention. | -- Typing bye() causes swirl to exit. Your progress will be saved. | -- Typing main() returns you to swirl's main menu. | -- Typing info() displays these options again. | Let's get started! sqrt() function and to take the absolute value, use the abs() function Vector of unequal length Artihmetic Op