It’s actually impossible to do so. The two softwares require the opposite of sysmtem configuration. On my Windows 10, I need to do two things to switch the envirenment so that either of them can work.

1. Hyper-V https://blogs.technet.microsoft.com/gmarchetti/2008/12/07/turning-hyper-v-on-and-off/

2. Memoery Isolation http://www.win10.today/jc/7941.html

Docker needs both features on while VMware needs both of them off.

TL;DR: Hard drive has physical partitions (physical volumes), which can be assigned to logical volume groups, which can be devided to logical volumes, which can be somehow mapped as disk, which can be mounted to linux directories. using an example:

/dev/sda(hard drive) -> /dev/sda2(partition) -> centos(volume group) -> /dev/centos/root -> /dev/mapper/centos-root -> /

and the commands used are:
fdisk -l -> vgdisplay -> lvdisplay -> df -h

I’d like to increase the general space of my vmware virtual host. After allocating more space (1GB) to the host by modifying the configuratiton on vmware, I feel confused.

When installing CentOS, I used its default setting. This is the current situation:

1
2
3
4
5
6
7
8
9
10
11

[root@cluster01 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   17G   14G  3.4G  81% /
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G     0  1.9G   0% /dev/shm
tmpfs                    1.9G   12M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1               1014M  142M  873M  14% /boot
tmpfs                    378M     0  378M   0% /run/user/0
tmpfs                    378M     0  378M   0% /run/user/998
cm_processes             1.9G     0  1.9G   0% /run/cloudera-scm-agent/process

As can be seen, whatever mapping to “/“ has limited space left.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

[root@cluster01 ~]# fdisk -l

Disk /dev/sda: 32.2 GB, 32212254720 bytes, 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a2f79

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200    41943039    19921920   8e  Linux LVM

Disk /dev/mapper/centos-root: 18.2 GB, 18249416704 bytes, 35643392 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

I am honestly dazzled by this info, what I can gather is:

1. Disk /dev/sda: 32.2 GB this looks like the size of the physical disk
2. /dev/sda2 2099200 41943039 19921920 8e Linux LVM this looks like a device using the physical disk
3. Disk /dev/mapper/centos-root: 18.2 GB whatever it is, this is what’s mapped to “/“ folder

At this point, I naturally want to move space from 1 to 3. After some googling, the following is what I do:

1. Create a new primary partition, making it a “linux lvm” type:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

[root@cluster01 ~]# fdisk /dev/sda
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   g   create a new empty GPT partition table
   G   create an IRIX (SGI) partition table
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): n
Partition type:
   p   primary (2 primary, 0 extended, 2 free)
   e   extended
Select (default p): p
Partition number (3,4, default 3):
First sector (41943040-62914559, default 41943040):
Using default value 41943040
Last sector, +sectors or +size{K,M,G} (41943040-62914559, default 62914559):
Using default value 62914559
Partition 3 of type Linux and of size 10 GiB is set

Command (m for help): t
Partition number (1-3, default 3):
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

What I do here is to create a new partition which takes all the free space, and then change its type to “8e”(code for “linux lv”)

1
2
3
4
5
6
7
8
9
10
11
12

[root@cluster01 ~]# fdisk -l

Disk /dev/sda: 32.2 GB, 32212254720 bytes, 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a2f79

/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200    41943039    19921920   8e  Linux LVM
/dev/sda3        41943040    62914559    10485760   8e  Linux LVM

2. After rebooting, allocating space to volume group

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

[root@cluster01 ~]# vgdisplay
  --- Volume group ---
  VG Name               centos
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <19.00 GiB
  PE Size               4.00 MiB
  Total PE              4863
  Alloc PE / Size       4863 / <19.00 GiB
  Free  PE / Size       0 / 0
  VG UUID               QrSMSB-dJyP-7JYM-GyAy-UKPW-0XBE-PqCYQb
  
[root@cluster01 ~]# vgextend centos /dev/sda3
  Physical volume "/dev/sda3" successfully created.
  Volume group "centos" successfully extended
  
[root@cluster01 ~]# vgdisplay
  --- Volume group ---
  VG Name               centos
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               28.99 GiB
  PE Size               4.00 MiB
  Total PE              7422
  Alloc PE / Size       4863 / <19.00 GiB
  Free  PE / Size       2559 / <10.00 GiB
  VG UUID               QrSMSB-dJyP-7JYM-GyAy-UKPW-0XBE-PqCYQb

3. Allocating free space from volume group to logical volume

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

[root@cluster01 ~]# lvdisplay
  --- Logical volume ---
  LV Path                /dev/centos/swap
  LV Name                swap
  VG Name                centos
  LV UUID                dbl59z-eFfF-DJ8G-QHKS-akGb-VBc2-ZQ93sD
  LV Write Access        read/write
  LV Creation host, time cluster01.john42.com, 2018-08-21 13:58:38 +0800
  LV Status              available
  # open                 2
  LV Size                2.00 GiB
  Current LE             512
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:1

  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  LV UUID                eiV78Y-pizG-9frt-y1ak-LTII-baUQ-NZhuJF
  LV Write Access        read/write
  LV Creation host, time cluster01.john42.com, 2018-08-21 13:58:38 +0800
  LV Status              available
  # open                 1
  LV Size                <17.00 GiB
  Current LE             4351
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0
  
[root@cluster01 ~]# lvextend -l +100%FREE /dev/centos/root
  Size of logical volume centos/root changed from <17.00 GiB (4351 extents) to 26.99 GiB (6910 extents).
  Logical volume centos/root successfully resized.
  
[root@cluster01 ~]# lvdisplay
  --- Logical volume ---
  LV Path                /dev/centos/swap
  LV Name                swap
  VG Name                centos
  LV UUID                dbl59z-eFfF-DJ8G-QHKS-akGb-VBc2-ZQ93sD
  LV Write Access        read/write
  LV Creation host, time cluster01.john42.com, 2018-08-21 13:58:38 +0800
  LV Status              available
  # open                 2
  LV Size                2.00 GiB
  Current LE             512
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:1

  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  LV UUID                eiV78Y-pizG-9frt-y1ak-LTII-baUQ-NZhuJF
  LV Write Access        read/write
  LV Creation host, time cluster01.john42.com, 2018-08-21 13:58:38 +0800
  LV Status              available
  # open                 1
  LV Size                26.99 GiB
  Current LE             6910
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0

4. resize the volume:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

[root@cluster01 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   17G   14G  3.4G  81% /
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G     0  1.9G   0% /dev/shm
tmpfs                    1.9G   12M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1               1014M  142M  873M  14% /boot
tmpfs                    378M     0  378M   0% /run/user/0
tmpfs                    378M     0  378M   0% /run/user/998
cm_processes             1.9G     0  1.9G   0% /run/cloudera-scm-agent/process

[root@cluster01 ~]# xfs_growfs /dev/centos/root
meta-data=/dev/mapper/centos-root isize=512    agcount=4, agsize=1113856 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=4455424, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 4455424 to 7075840

[root@cluster01 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   27G   14G   14G  51% /
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G     0  1.9G   0% /dev/shm
tmpfs                    1.9G   12M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1               1014M  142M  873M  14% /boot
tmpfs                    378M     0  378M   0% /run/user/0
tmpfs                    378M     0  378M   0% /run/user/998
cm_processes             1.9G     0  1.9G   0% /run/cloudera-scm-agent/process

This is a listing of all the work needed for installing CM and CDH5 on CentOS 7 for study purpose. All servers are virtually hosted on VMware Workstation.

A few words before I begin. I find the official documents both informative and confusing at the same time, partially owing to the fact that there are
several paths to installation and these paths diverge at one time and overlapse at another time. One gotta makes sure to stay within the chosen path, otherwise one may end up tripping down the rabbit hole. Some other confusion results from lacking a big picture of different roles played by vairious components/softwares. Diving in the details of the documents proves not the best choice I’ve made. Anyway, if ever I install the system again, the first two documents I should read are:

• https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cm_ig_intro_to_cm_install.html
• https://www.cloudera.com/documentation/enterprise/5-5-x/topics/installation_installation.html#concept_qpf_2d2_2p

TL;DR:
The path I choose is to install Cloudera Manager first, and let it install other components automatically for me. At the same time, I prefer offline repositories for all dependencies. Also, between a “Demonstration and proof of concept deployments” and a “Production deployments” ref, I prefer the latter.

Preparetion

SSH Setup

To facilitate fast deployment, master host should be able to get access to other hosts with ssh, password free:

• ssh-keygen -t rsa
• ssh-copy-id cluster0[2-4]

Networks

I mainly use nmtui tool to configure the network.

let the hosts file speaks

1
2
3
4
5
6
7

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.146.101 cluster01.john.how cluster01
192.168.146.102 cluster02.john.how cluster02
192.168.146.103 cluster03.john.how cluster03
192.168.146.104 cluster04.john.how cluster04

make sure all hosts have a copy:

1

scp /etc/hosts @cluster02:/etc/

Security & Firewall

shut ‘em down.

• shutdown firewall (or iptables)

  1 2	systemctl stop firewalld.service systemctl disable firewalld.service

• modify /etc/selinux/config -> SELINUX=disabled (reboot needed)

Repositories

Since all servers cannot connect to the internet (when they do, it’s slow anyway), I’m going with the offline route.

Parcels

Accoding to the official documents, “For a parcel installation, only the Cloudera Manager Server needs Internet access, but for a package installation, all cluster hosts require access to the Cloudera repository”. Although my master server has internet access, I’m going to set up an internal parcel repository anyway, so that I can utilize some downloader software to expedite the downloading process.

You can find the detailed steps here.

The abreviated steps are :

1. Set up an Apache Http Server

   1 2	yum install httpd service httpd start

2. Download the parcel files for CentOS from https://archive.cloudera.com/cdh5/parcels/5.15/
   These are the files I think relavent for me
   • CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel
   • CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1
   • manifest.json
3. Move files to server directory:

   1 2 3 4

   mkdir -p /var/www/html/cloudera-parcels/cdh5/<cdh5_version>/ mv *.parcel* /var/www/html/cloudera-parcels/cdh5/<cdh5_version>/ mv manifest.json /var/www/html/cloudera-parcels/cdh5/ chmod -R ugo+rX /var/www/html/cloudera-parcels/cdh5/<cdh5_version>/

4. Verify the server by browsing http://<web_server>/cdh5/<cdh5_version>

Package

I am not clear at the moment that if both Parcel and Package are needed, but since install the Oracle Java JDK needs the package, I’m going to set up the offline repository (Since the requested http server has been established, I’m going to skip those steps)

1. Download the tarball: https://archive.cloudera.com/cm5/repo-as-tarball
2. tar xvfz cm5.15.1-centos7.tar.gz
3. mv cm /var/www/html
4. chmod -R ugo+rX /var/www/html/cm
5. To verify, navigate to http://hostname:port/cm
6. Config yum repo file: vi /etc/yum.repos.d/myrepo.repo

   1 2 3 4 5

   [myrepo] name=myrepo baseurl=http://cluster01.localdomain/cm/5.15.1 enabled=1 gpgcheck=0

   copy to every host: scp /etc/yum.repos.d/myrepo.repo cluster02:/etc/yum.repos.d/

MySQL

This part has been lost… Reference here
One thing to note is one should NOT use the lastest version, since JDK 7 will be installed (See below) by default and the latest connector requires JRE 1.8.

Java JDK

yum install oracle-j2sdk1.7

Installation

Having jumped all the loops and hoops, I can finally get down to the real business:

Install the Cloudera Manager Server Packages

yum install cloudera-manager-daemons cloudera-manager-server

Preparing a Cloudera Manager Server External Database

/usr/share/cmf/schema/scm_prepare_database.sh mysql [-uuser -p] scm scm password

Start the Cloudera Manager Server

service cloudera-scm-server start

Start and Log into the Cloudera Manager Admin Console

1. Look for errors tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
2. Navigate to http://Server host:7180
3. Login admin admin

Installation Process

1. After a few Agreements and edition selection, I’m asked to choose the hosts. And I input cluster0[1-3], leaving cluster04 out of it (saving it for future)
2. When it comes to parcels, select “more options” and input the local parcels server address http://192.168.146.101/cloudera-parcels/cdh5/5.15.1/ (possibly need to move menifest.json to this folder)
3. A bunch of self-explainary options
4. The critical step is to tell the installer to use your own repository:http://192.168.146.101/cm/5.15.1/

Troubleshooting

• Nodemananger start failure: run this command on every Nodemanager host chmod -R 755 /var/lib/hadoop-yarn/

Reference

• The Official
• The Official Before You Install
• Random offline installation guide (https://www.zybuluo.com/sasaki/note/242142)
• not directly related to installation per se, but gives you a general idea of how many servers needed and what roles they each bear Recommended Cluster Hosts and Role Distribution

sudo yum --nogpgcheck localinstall oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

Simply put, @EnableConfigurationProperties is acted like a general switch, enabling or disabling (by absense) Spring Boot’s functionality to bind a certain properties from the environment to a pojo class, and @ConfigurationProperties is used on the said pojo, as the receiver of environmental properties.

These two annotations should be both used at the same time in combinations in a project, as follows:

1. on the app’s configuration class:

   1 2 3

   @SpringBootApplication @EnableConfigurationProperties public class SpringBootWebApplication {...}

   and one the pojo

   1 2 3

   @Component @ConfigurationProperties public class TestConfig {}

In this case, @EnableConfigurationProperties enables the functionality app-wise, and the config pojo registers itself with the help of @Component.

2.
on the app’s configuration class:

1
2
3

@SpringBootApplication
@EnableConfigurationProperties(TestConfig.class)
public class SpringBootWebApplication {...}

and one the pojo

1
2

@ConfigurationProperties
public class TestConfig {}

In this case, @EnableConfigurationProperties not only enables the functionality, but also selects a config pojo to be used in this project. The config pojo (TestConfig) doesn’t have to register itself through @Component.

Scenario:

Spring MVC application, with three tiers from bottom up: DAO, Service and Controller.

Things to consider:

• In principle, unchecked exceptions indicate bugs, and checked exceptions indicate problems outside your control1.
• Checked exception is a unique feature only in Java. It’s experimental and hard to get right, to put it anther way, people have controversial opinions on what’s right.
• Unchecked exceptions are like exceptions in other languages, such as in JavaScript or Python.
• One can think of checked exceptions as an alternative type that a method can return. Together with the normal type, exceptions form a union type.

Patterns

Anti-patterns

• Catch an exception, log it, and rethrow it (or wrap it in a new exception), cuz “This is cumbersome, error prone (it’s easy to lose the stacktrace) and serves no useful purpose”[2].

Goal/Requirements:

• required Keep app from crashing, ie exceptions don’t bubble up to users in a raw state (jsp error page with stacktrace, or api not returning intentional and predefined result (resultCode, message))
• required Make sure exceptions get logged, preferrably:
  • logged once
  • with previous cause
  • with parameter map
• optional Keep influences on business logic minumum, ie clean code
• optional Keep concerns separated
• required Keep checked exceptions consistent with the tiers throwing them, eg. all classes in DAO throw DAOException, and all classes in service throw ServiceExceptiopn. This also helps with a stable signature of method, otherwise, when the underlining implement changes, the signature might also need to change accordingly.

Design Decisions

• Validate parameters at the beginning of a method, and throw RuntimeExceptions if violation is found. Or use assertions that throw RuntimeExceptions. It’s the responsibility of callers to make sure all parameters are legal.
• Must catch exceptions in controllers (or same tier)
  • Use cross-cut exception handlers if possible
    • how to return error to pages in handlers?

Relative

• Handling exceptions in a multithread application

Reading materials:

• Effective Java Exception
• Does Java need Checked Exceptions?

[2]:Expert One-on-one J2EE Design And Development by Rod Johnson

SQL

• SQL Performance Explained