Posts

Showing posts from September, 2018

Install wget on Windows

Install Cholatey https://chocolatey.org/ Start an elevated administrative command shell and run this command choco install wget

Angular 6 tutorial by

Image
Followed Part 1 of the Angular 6 tutorial here https://github.com/techiediaries/angular-express-jwt Documenting the Postman settings here as they were not covered in the tutorial Get the secret access token no settings needed in Authorization tab No settings needed in Headers tab Pass the email and password in the Body tab Results after I hit the "Send" button. Using the secret access token to submit a new record Authorization tab- no settings required Headers tab - need to add in the Authorization key and  set  value as "Bearer <access token>" Body tab pass in the new record as json Successfully added the record And some settings for ease of copy-and-paste http://localhost:3000/auth/login {"id":2,"username":"Gertrude25","password":"ytoC9PpKFDiJiBf","email":"Molly.Barrows@hotmail.com"} access token Bearer <accesstoken> h

Finally Spark 2 runs on CDH 5.15! :)

Image
These are the things that I needed to setup or troubleshoot before it got running: Upgrade Cloudera Manager  to the latest version by following the instructions on this website https://www.cloudera.com/documentation/enterprise/5-13-x/topics/cm_ag_upgrading_cm.html Note: to get Cloudera Manager Express to run on VM with less memory, use --force sudo /home/cloudera/cloudera-manager --pause --express --force The important steps would be as follows:- 1) Download cloudera-manager.repo with  wget https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo 2)Put cloudera-manager.repo inside  /etc/yum.repos.d/ 3) Run these commands sudo yum clean all sudo yum upgrade cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent Upgrade JDK 1.7 to 1.8 Download JDK 1.8 or latest version of JDK and unzip to chosen folder. I put mine in /usr/java/jdk1.8 Set the JAVA_HOME path in .bashrc Set the JAVA_HOME in Cloudera Managers->Hosts->Conf

Linux Bash scripts

sudo nano ~/.bashrc source ~/.bashrc

Upgrade Cloudera manger

Note on 07-Sep-2018: to try another day https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ag_upgrade_cm5.html#concept_tsy_5vq_wn

Upgrade Spark

Follow instructions here https://www.cloudera.com/documentation/spark2/2-1-x/topics/spark2_installing.html Note: you may need to also install newer version of CDH 5.13 5.13.0-1.cdh5.13.0.p0.29 Download CSD http://archive.cloudera.com/spark2/csd/ Place this in /opt/cloudera/csd Set ownership to  cloudera-scm:cloudera-scm  with permission 644. chown     cloudera-scm:cloudera-scm   *.jar sudo  chmod 644 *.jar Restart Cloudera SCM service cloudera-scm-server restart Download Parcel https://archive.cloudera.com/spark2/parcels/latest/ Add Parcel repositories to the Configuration section, download and distribute http://quickstart.cloudera:7180/cmf/parcel/status
Image
Trying out Cloudera Express http://quickstart.cloudera:7180

Useful HDFS commands

Remove directories recursively hdfs dfs rm -r /loudacre/webpage_files References https://dzone.com/articles/top-10-hadoop-shell-commands

mysql commands for working with the Cloudera VMs

Login mysql -uroot -pcloudera Dump a database to a file mysqldump -u root -p mydatabase > /home/myuser/database-dump.sql Restore mySQL dump file to a mySQL database mysql -u username -p database_name < file.sql mysqldump -u training -p trainingdatabase > /home/training/trainingdatabase-dump.sql mysql -u root -p loudacre < loudacredatabase-dump.sql mySQL commands SHOW DATABASES; CREATE DATABASE loudacre; use loudacre; show tables; mysql -u root -p loudacre -e "describe webpage" mysql -u root -p loudacre -e "select * from webpage limit 5"