Posts

Showing posts from October, 2018

numpy find substring

arraythatfitsthecriteria = np.core.defchararray.startswith(nparray, substringtofind)

Useful Raspbian commands

Check free disk space df -h Install MySQL Server sudo apt-get install mysql-server sudo mysql_secure_installation sudo mysql -u root -p sudo apt-get install python-mysqldb sudo mysql -u root -p GRANT ALL PRIVILEGES ON mydb.* TO 'username'@'localhost' IDENTIFIED BY 'password'; https://pimylifeup.com/raspberry-pi-mysql-phpmyadmin/ Install myPHPAdmin sudo apt-get install phpmyadmin Restart Apache server sudo service apache2 restart Restart MySQL server sudo service mysql restart Change file permissions sudo chmod -R 777 phpmyadmin Edit phpmyadmin blowfish configuration file sudo nano /var/lib/phpmyadmin/blowfish_secret.inc.php sudo nano /etc/phpMyAdmin/config.inc.php

Code Editors

Python Survey 2017 Aptana Atom Eclipse + Pydev Emacs Gedit IDLE IntelliJ IDEA Jupyter Notebook Komodo Editor Komodo IDE NetBeans Ninja-IDE NotePad++ PyCharm Community Edition PyCharm Professional Edition Python Tools for Visual Studio (PTVS) Rodeo Spyder Sublime Text TextMate Vim VS Code Wing IDE

onfiguration management tools d

Ansible Chef Puppet Salt

Continuous Integration (CI) system(s)

TeamCity Jenkins / Hudson Gitlab CI CruiseControl Travis CI AppVeyor Bamboo CircleCI

Big Data tools

Apache Hadoop/MapReduce Apache Tez ClickHouse Apache Samza Apache Hive Dask Apache Spark Apache Beam Apache Kafka Apache Flink

Databases

PostgreSQL HBase Amazon Redshift MongoDB DB2 Neo4j Cassandra Couchbase MySQL h2 Redis SQLite MS SQL Server Oracle Database

Python ORM

Dejavu Tortoise ORM PonyORM SQLObject Django ORM Peewee SQLAlchemy

Woring with datetime objects in Numpy and Pandas and core Python

import datetime e = np.array(['2017-01-01', '2017-01-02', '2017-01-03'],dtype='datetime64') t = e[0] day = t.astype(datetime.datetime).isoweekday()

Saving data to MySQL with Python

import mysql.connector from time import sleep import random import sys ## C:\ProgramData\MySQL\MySQL Server 8.0\my.ini ## https://dev.mysql.com/doc/refman/8.0/en/caching-sha2-pluggable-authentication.html#caching-sha2-pluggable-authentication-installation try:     username = "yourusername"; password="yourpassword";     database="yourdatabase";host="localhost";     cnx = mysql.connector.connect(user=username, password=password,                                   host=host,database=database)     cursor = cnx.cursor()     print("Successfully connected to database!")          update = True     while update:         try:             value = random.randint(0,1024)              sensor_value = value             print("Sensor value:", sensor_value)             insert_statement = "INSERT INTO lights (light_value) VALUES (%                                       (light_value)s)"             data = { 'l

Install gspread with Anaconda

Use this command conda install -c conda-forge gspread  Read about it at https://anaconda.org/conda-forge/gspread More links about gspread below https://pypi.org/project/gspread/ https://github.com/burnash/gspread https://gspread.readthedocs.io/en/latest/

Writing a GenericEncoder to prevent JSON conversion errors

class GenericEncoder ( json.JSONEncoder ): import numpy def default ( self , obj ) : if isinstance (obj, numpy.generic): return numpy.asscalar(obj) elif isinstance (obj, datetime.datetime): return obj.strftime( '%Y-%m-%d %H:%M:%S' ) else : return json.JSONEncoder.default( self , obj)

How to convert a pandas GroupBy to a DataFrame

# Create a new DataFrameGroupBy object that groups the data by registration year group_by_years = df.groupby(df[ "month" ].dt.year) # Let's count how many rows are there in each year group_by_years_count = group_by_years.count() # We have too many columns to display, let's reduce the number of columns to just one group_by_years_count = group_by_years_count[[ "month" ]] # rename the column so it looks more proper on the barplot later group_by_years_count.rename( columns = { 'month' : 'count' }, inplace = True ) group_by_years_count.reset_index( inplace = True ) df_new = group_by_years_count.apply( list ).apply(pd.Series) df_new.columns = [ 'month' , 'count' ]

Using MySQL Connector/Python 8.0 with MySQL 8.0

https://insidemysql.com/using-mysql-connector-python-8-0-with-mysql-8-0/ https://stackoverflow.com/questions/35963592/python3-mysql-ssl-error-mysql-connector https://github.com/PyMySQL/mysqlclient-python

Upgrade Python on Anaconda

https://conda.io/docs/user-guide/tasks/manage-python.html#updating-or-upgrading-python https://www.anaconda.com/download/ Upgrade to the latest version on the same branch conda update python Upgrade to a specific version conda install python = 3 .6

Spark Streaming

Spark Streaming https://dzone.com/articles/cassandra-sink-for-spark-structured-streaming

Set time on Raspberry Pi

sudo timedatectl set-time '2018-10-02 09:53' date -s "2 OCT 2006 18:00:00" timedatectl set-time 04:00:00 sudo timedatectl set-time 04:00:00  sudo timedatectl set-ntp false  sudo timedatectl set-time 04:00:00  sudo timedatectl timedatectl set-ntp yes  yum install ntp ntpdate # systemctl start ntpd # systemctl enable ntpd # systemctl status ntpd # ntpdate -u -s 0.centos.pool.ntp.org 1.centos.pool.ntp.org 2.centos.pool.ntp.org Next, restart the ntpd daemon to synchronize CentOS NTP server date and time with your local date and time. # systemctl restart ntpd Now check using the timedatectl command if NTP synchronization is enabled and if it is actually synchronized. # timedatectl Lastly, using the hwclock utility, set the hardware clock to the current system time using the -w flag as follows. # hwclock  -w timedatectl list-timezones tzselect You can make this change permanent for yourself by appending the line         TZ='Asia/Singapore