Ashwani's Tech World - Giving back to community

SPIFFE Implementation with SPIRE And OIDC Setup

2023-02-26T18:32:22+00:00

As a solution architect, I'm excited to share my thoughts on two important frameworks in the world of technology - SPIFFE and SPIRE. These frameworks have been gaining popularity in last few years, and they play a crucial role in zero trust and ensuring the security and privacy of data in various industries.

In this blog, I'll not explore the concepts behind SPIFFE and SPIRE and their benefits for businesses. I'll also not discuss how these frameworks work together to provide a comprehensive security solution. Please Read more about SPIFFE and SPIRE seperately. This article is a note to myself how to quickstart a demo for SPIRE and implementation of SPIFFE.

Configure SPIRE Server on Kubernetes

Clone https://github.com/spiffe/spire-tutorials and obtain the .yaml files from the spire-tutorials/k8s/quickstart subdirectory.

Remember to run all kubectl commands in the directory in which those files reside.

Configure Kubernetes Namespace for SPIRE Components

Follow these steps to configure the spire namespace in which SPIRE Server and SPIRE Agent are deployed.

Change to the directory containing the .yaml files.

Create the namespace:

$ kubectl apply -f spire-namespace.yaml

Run the following command and verify that spire is listed in the output: $ kubectl get namespaces

Create Server Bundle Configmap, Role & ClusterRoleBinding

For the server to function, it is necessary for it to provide agents with certificates that they can use to verify the identity of the server when establishing a connection.

In a deployment such as this, where the agent and server share the same cluster, SPIRE can be configured to automatically generate these certificates on a periodic basis and update a configmap with contents of the certificate. To do that, the server needs the ability to get and patch a configmap object in the spire namespace.

To allow the server to read and write to this configmap, a ClusterRole must be created that confers the appropriate entitlements to Kubernetes RBAC, and that ClusterRoleBinding must be associated with the service account created in the previous step.

Create the server's service account, configmap and associated role bindings as follows:

$ kubectl apply \
    -f server-account.yaml \
    -f spire-bundle-configmap.yaml \
    -f server-cluster-role.yaml

Create Server Configmap

The server is configured in the Kubernetes configmap specified in server-configmap.yaml, which specifies a number of important directories, notably /run/spire/data and /run/spire/config. These volumes are bound in when the server container is deployed.

Change service type to LoadBalancer instead of NodePort in the server-service.yaml Add NodeAttestor join_token in server-configmap.yaml

    NodeAttestor "join_token" {
        plugin_data {}
      }

Deploy the server configmap and statefulset by applying the following files via kubectl:

$ kubectl apply \
    -f server-configmap.yaml \
    -f server-statefulset.yaml \
    -f server-service.yaml

This creates a statefulset called spire-server in the spire namespace and starts up a spire-server pod, as demonstrated in the output of the following commands:

$ kubectl get statefulset --namespace spire

NAME           READY   AGE
spire-server   1/1     86m

$ kubectl get pods --namespace spire

NAME                           READY   STATUS    RESTARTS   AGE
spire-server-0                 1/1     Running   0          86m

$ kubectl get services --namespace spire

NAME           TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
spire-server   NodePort   10.107.205.29           8081:30337/TCP   88m

Create Join Token

Generate a one-time-use token to use to attest the agent:

On Server

Creating a join token to attest the agent to the server A join token is one of the many available agent attestor methods. It is a one-time-use, pre-shared key that attests (authenticates) the SPIRE agent to the SPIRE server. Other agent attestation methods include AWS/GCP instance identity tokens and X.509 certificates. To see a complete list of available attestors, click https://deploy-preview-272--spiffe.netlify.app/docs/latest/spire/using/registering/#1-defining-the-spiffe-id-of-the-agent.

kubectl -n spire exec -it $(kubectl -n spire get pods -o=jsonpath='{.items[0].metadata.name}' -l app=spire-server) -- /opt/spire/bin/spire-server token generate  -spiffeID spiffe://example.org/myagent

Token: db2exxxxx-61d6-428e-8297-1efc0xxxxxx

Make a note of the token, you will need it in the next step to attest the agent on initial startup.

A Join Token is just one of the many available agent attestation methods. To see a complete list of available attestors, click https://deploy-preview-272--spiffe.netlify.app/docs/latest/spire/using/registering/#1-defining-the-spiffe-id-of-the-agent.

Configure Agent on client side (on-prem)

Install agent

Read this: https://deploy-preview-272--spiffe.netlify.app/docs/latest/try/getting-started-linux-macos-x/#downloading-spire-for-linux $ curl -s -N -L https://github.com/spiffe/spire/releases/download/v1.4.7/spire-1.4.7-linux-x86_64-glibc.tar.gz | tar xz Create, and move the SPIRE executables into, a bin directory as expected by the rest of this document: $ mkdir bin $ mv spire-server spire-agent bin

Starting the SPIRE Agent

SPIRE agents query the SPIRE server to attest (authenticate) nodes and workloads.

Use the token created in the previous step to start and attest the agent:

$ spire-agent run -config /opt/spire/conf/agent/agent.conf -joinToken xxxxxc5-4336-456f-8140-05850xxxxxxxx

WARN[0000] Current umask 0022 is too permissive; setting umask 0027
INFO[0000] Starting agent with data directory: "./data/agent"
INFO[0000] Plugin loaded                                 external=false plugin_name=disk plugin_type=KeyManager subsystem_name=catalog
INFO[0000] Plugin loaded                                 external=false plugin_name=join_token plugin_type=NodeAttestor subsystem_name=catalog
INFO[0000] Plugin loaded                                 external=false plugin_name=unix plugin_type=WorkloadAttestor subsystem_name=catalog
INFO[0000] Bundle is not found                           subsystem_name=attestor
DEBU[0000] No pre-existing agent SVID found. Will perform node attestation  subsystem_name=attestor
WARN[0000] Keys recovered, but no SVID found. Generating new keypair  subsystem_name=attestor
INFO[0000] SVID is not found. Starting node attestation  subsystem_name=attestor
WARN[0000] Insecure bootstrap enabled; skipping server certificate verification  subsystem_name=attestor
INFO[0000] Node attestation was successful               rettestable=false spiffe_id="spiffe://example.org/spire/agent/join_token/b4ef1fc5-4336-456f-8140-05xxxxxxx335e" subsystem_name=attestor
DEBU[0001] Entry created                                 entry=9638xxxx-a983-499a-89ea-a6cxxxxxx385b selectors_added=1 spiffe_id="spiffe://example.org/myagent" subsystem_name=cache_manager
DEBU[0001] Renewing stale entries                        cache_type=workload count=1 limit=500 subsystem_name=manager
INFO[0001] Renewing X509-SVID                            spiffe_id="spiffe://example.org/myagent" subsystem_name=manager
DEBU[0001] SVID updated                                  entry=963846e0-a983-499a-89ea-a6c78f3f385b spiffe_id="spiffe://example.org/myagent" subsystem_name=cache_manager
DEBU[0001] Bundle added                                  subsystem_name=svid_store_cache trust_domain_id=example.org
INFO[0001] Starting Workload and SDS APIs                address=/tmp/spire-agent/public/api.sock network=unix subsystem_name=endpoints
DEBU[0002] Starting checker                              name=agent subsystem_name=health

Check logs on server

kubectl logs -n spire $(kubectl -n spire get pods -o=jsonpath='{.items[0].metadata.name}' -l app=spire-server) -f 
....
....
time="2023-02-26T14:48:10Z" level=debug msg="Initializing health checkers" subsystem_name=health
time="2023-02-26T14:48:10Z" level=info msg="Serving health checks" address="0.0.0.0:8080" subsystem_name=health
time="2023-02-26T15:08:16Z" level=error msg="Invalid argument: join token expired" authorized_as=nobody authorized_via= caller_addr="10.56.1.1:27045" method=AttestAgent node_attestor_type=join_token request_id=3c3c4446-89aa-41f5-9f2b-77e86faeceba service=agent.v1.Agent subsystem_name=api
time="2023-02-26T15:08:56Z" level=info msg="Agent attestation request completed" address="10.128.0.5:41034" agent_id="spiffe://example.org/spire/agent/join_token/b4xxxxx5-4336-456f-8140-xxxxe9335e" authorized_as=nobody authorized_via= caller_addr="10.128.0.5:41034" method=AttestAgent node_attestor_type=join_token request_id=2f2e41bb-1f9d-4ddc-95c5-xxxxxxdd95 service=agent.v1.Agent subsystem_name=api
time="2023-02-26T15:08:57Z" level=debug msg="Signed X509 SVID" authorized_as=agent authorized_via=datastore caller_addr="10.56.1.1:49076" caller_id="spiffe://example.org/spire/agent/join_token/b4ef1fc5-4336-456f-8140-05xxxxxx9335e" entry_id=9638xxxx-a983-499a-89ea-a6c78xxxxx85b expiration="2023-02-26T16:08:57Z" method=BatchNewX509SVID request_id=bec5f734-70b3-4e1a-8562-c5ffefce7c1a service=svid.v1.SVID spiffe_id="spiffe://example.org/myagent" subsystem_name=api

Register Workload

On Server

Create a registration policy for your workload In order for SPIRE to identify a workload, you must register the workload with the SPIRE Server, via registration entries. Workload registration tells SPIRE how to identify the workload and which SPIFFE ID to give it.

This command is creating a registration entry based on the current user’s UID ($(id -u)) - feel free to adjust this as necessary

$ kubectl -n spire exec -it $(kubectl -n spire get pods -o=jsonpath='{.items[0].metadata.name}' -l app=spire-server) -- bin/spire-server entry create -parentID spiffe://example.org/myagent  -spiffeID spiffe://example.org/myservice -selector unix:uid:$(id -u)

Entry ID      : ac5e2354-596a-4059-85f7-5b7xxxxxxxxx
SPIFFE ID     : spiffe://example.org/myservice
Parent ID     : spiffe://example.org/myagent
TTL           : 3600
Selector      : unix:uid:501

On Agent

Retrieve and view a x509-SVID This command replicates the process that a workload would take to get an x509-SVID from the agent. The x509-SVID could be used to authenticate the workload to another workload. To fetch and write an x509-SVID to /tmp/:

$ spire-agent api fetch x509 -write /tmp/

Received 1 svid after 2.2703ms

SPIFFE ID:              spiffe://example.org/myservice
SVID Valid After:       2023-02-26 15:18:53 +0000 UTC
SVID Valid Until:       2023-02-26 16:19:03 +0000 UTC
CA #1 Valid After:      2023-02-26 14:47:58 +0000 UTC
CA #1 Valid Until:      2023-02-27 14:48:08 +0000 UTC

Writing SVID #0 to file /tmp/svid.0.pem.
Writing key #0 to file /tmp/svid.0.key.
Writing bundle #0 to file /tmp/bundle.0.pem.

Check logs on Server

Logs: kubectl logs -n spire $(kubectl -n spire get pods -o=jsonpath='{.items[0].metadata.name}' -l app=spire-server) -f

time="2023-02-26T15:19:03Z" level=debug msg="Signed X509 SVID" authorized_as=agent authorized_via=datastore caller_addr="10.56.1.1:28674" caller_id="spiffe://example.org/spire/agent/join_token/xxxx336-456f-8140-05850ae9335e" entry_id=e1d7a74d-56b9-4fc2-b5de-0ff48xxx629 expiration="2023-02-26T16:19:03Z" method=BatchNewX509SVID request_id=d36dxxx4bb-466b-afe7-b0f047156317 service=svid.v1.SVID spiffe_id="spiffe://example.org/myservice" subsystem_name=api

You can use the openssl command to view the contents of the SVID:

openssl x509 -in /tmp/svid.0.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            92:23:22:2c:6b:15:84:f7:30:xx:xx:xx:xx:3e:b9:08
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = SPIFFE
        Validity
            Not Before: Feb 26 15:18:53 2023 GMT
            Not After : Feb 26 16:19:03 2023 GMT
        Subject: C = US, O = SPIRE, x500UniqueIdentifier = f7fa051b660xxxx34e54056a8c4e281a
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:44:33:2a:c1:af:c0:5c:77:58:7f:a9:fb:3b:41:
                    d0:bc:xy:xx:21:4e:d8:13:65:25:30:24:48:6f:b6:
                    0b:84:73:5a:xx:xx:72:52:c0:6a:48:11:a1:72:94:
                    f4:31:87:cf:6d:9f:xx:91:83:0b:e3:d1:0f:88:1b:
                    46:e4:49:1c:b7
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                51:FD:A9:DA:35:51:10:65:xx:xx:xx:01:F6:28:C9:34:CF:07:C5:11
            X509v3 Authority Key Identifier:
                keyid:E1:61:D6:62:08:54:xx:xx:xx:C9:D9:50:2D:0F:13:EA:72:95:C1:34

            X509v3 Subject Alternative Name:
                URI:spiffe://example.org/myservice
    Signature Algorithm: sha256WithRSAEncryption
         1a:35:d6:47:ce:64:d0:85:2b:e4:76:b3:4e:0d:c5:db:f7:6b:
         32:fd:ac:a2:b4:f0:e7:77:bf:9c:3b:f8:b6:06:77:c3:9b:68:
         55:0b:5b:0b:72:55:ef:8c:dc:84:e0:b4:5a:78:79:7d:a3:db:
         b4:05:b3:c3:b6:c4:0c:46:f0:e0:eb:c9:64:df:75:2a:0c:9d:
         4c:dc:4c:25:58:ab:84:75:a9:49:cc:52:2f:04:72:46:8c:a1:
         e9:84:3c:7a:f4:a8:cd:4c:76:52:5f:34:13:28:xx:xx:xx:xx:
         18:f0:3a:b1:de:27:7c:7f:3a:97:9c:64:d8:6b:cf:ea:e6:d9:
         67:b6:1f:9c:54:7e:28:31:0b:bf:6c:e3:5d:3f:xx:xx:xx:xx:
         a4:a4:36:60:c1:87:60:39:3e:79:3a:5b:40:54:3c:12:48:fb:
         7a:5e:e1:9d:5a:c6:34:b4:e4:d0:1a:32:bc:10:7c:27:xx:xx:
         4d:8e:ae:b1:ee:de:3d:d3:bd:4c:ad:5d:cd:90:e9:be:c3:60:
         68:56:91:50:e4:ee:dd:8d:4e:e6:b8:cc:f3:2c:b5:c9:a7:7a:
         25:d3:9b:3e:ce:43:5f:93:57:3c:9a:59:d0:61:87:5e:1d:9d:
         c5:4c:42:c6:a2:7c:28:3a:a8:99:85:3c:84:05:c2:c9:1e:e3:
         22:b5:85:17

SETUP OIDC

https://deploy-preview-272--spiffe.netlify.app/docs/latest/keyless/vault/readme/

Make changes to files mentioned below under directory spire-tutorials/k8s/oidc-vault/k8s

oidc-dp-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: oidc-discovery-provider
  namespace: spire
data:
  oidc-discovery-provider.conf: |
    log_level = "INFO"
    # TODO: Replace MY_DISCOVERY_DOMAIN with the FQDN of the Discovery Provider that you will configure in DNS
    domains = ["spire-oidc.myddns.me"] 
    acme {
        directory_url = "https://acme-v02.api.letsencrypt.org/directory"
        cache_dir = "/run/spire"
        tos_accepted = true
        # TODO: Change MY_EMAIL_ADDRESS with your email
        email = "xxxx@gmail.com"
    }
    server_api {
      address = "unix:///tmp/spire-server/private/api.sock"
    }
    health_checks {}

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: spire-ingress
  namespace: spire
spec:
  tls:
    - hosts:
        # TODO: Replace MY_DISCOVERY_DOMAIN with the FQDN of the Discovery Provider that you will configure in DNS
        - spire-oidc.myddns.me
      secretName: oidc-secret
  rules:
    # TODO: Replace MY_DISCOVERY_DOMAIN with the FQDN of the Discovery Provider that you will configure in DNS
    - host: spire-oidc.myddns.me
      http:
        paths:
          - path: /.well-known/openid-configuration
            pathType: Prefix
            backend:
              service:
                name: spire-oidc
                port:
                  number: 443
          - path: /keys
            pathType: Prefix
            backend:
              service:
                name: spire-oidc
                port:
                  number: 443

server-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: spire-server
  namespace: spire
data:
  server.conf: |
    server {
      bind_address = "0.0.0.0"
      bind_port = "8081"
      socket_path = "/tmp/spire-server/private/api.sock"
      trust_domain = "example.org"
      data_dir = "/run/spire/data"
      log_level = "DEBUG"
      federation {
        bundle_endpoint {
          address = "0.0.0.0"
          port = 8443
        }
      }
      ca_key_type = "rsa-2048"

      # Creates the iss claim in JWT-SVIDs.
      # TODO: Replace MY_DISCOVERY_DOMAIN with the FQDN of the Discovery Provider that you will configure in DNS
      jwt_issuer = "spire-oidc.myddns.me"

      ca_subject = {
        country = ["US"],
        organization = ["SPIFFE"],
        common_name = "",
      }
    }

    plugins {
      DataStore "sql" {
        plugin_data {
          database_type = "sqlite3"
          connection_string = "/run/spire/data/datastore.sqlite3"
        }
      }

      NodeAttestor "k8s_sat" {
        plugin_data {
          clusters = {
            # TODO: Change this to your cluster name
            "spire-server" = {
              use_token_review_api_validation = true
              service_account_allow_list = ["spire:spire-agent"]
              service_account_allow_list = ["spire:spire-oidc"]
            }
          }
        }
      }

      KeyManager "disk" {
        plugin_data {
          keys_path = "/run/spire/data/keys.json"
        }
      }

      Notifier "k8sbundle" {
        plugin_data {
        }
      }
    }

    health_checks {
      listener_enabled = true
      bind_address = "0.0.0.0"
      bind_port = "8080"
      live_path = "/live"
      ready_path = "/ready"
    }

server-statefulset.yaml

Change image version to 1.5.4 and readinessProbe to /ready in server-statefulset.yaml

 - name: spire-oidc
          image: ghcr.io/spiffe/oidc-discovery-provider:1.5.4
          args:
          .
          .
          .
          readinessProbe:
            httpGet:
              path: /ready # TODO: Change this to /ready when using 1.5.2+
              port: 8008

Deploy the OIDC Discovery Provider Configmap

The SPIRE OIDC Discovery Provider provides a URL to the location of the Discovery Document specified by the OIDC protocol. The oidc-dp-configmap.yaml file specifies the URL to the OIDC Discovery Provider.

Before running the command below, ensure that you have replaced the MY_DISCOVERY_DOMAIN placeholder with the FQDN of the Discovery Provider as described in Replace Placeholder Strings in YAML Files.

Change to the directory k8s/oidc-vault/k8s containing the YAML files that describe the Kubernetes deployment and use the following command to apply the updated server ConfigMap, the ConfigMap for the OIDC Discovery Provider, and deploy the updated spire-server StatefulSet:

$ kubectl apply \
    -f server-configmap.yaml \
    -f oidc-dp-configmap.yaml \
    -f server-statefulset.yaml

To verify that the spire-server pod has spire-server and spire-oidc containers, run:

$ kubectl get pods -n spire -l app=spire-server -o \
    jsonpath='{.items[*].spec.containers[*].name}{"\n"}'


This should output:

spire-server spire-oidc

Configure the OIDC Discovery Provider Service and Ingress

Use the following command to set up a Service definition for the OIDC Discovery Provider and to configure an Ingress for that Service: ``` $ kubectl apply \ -f server-oidc-service.yaml \ -f ingress.yaml



### Part 2: Configure DNS for the OIDC Discovery IP Address

https://deploy-preview-272--spiffe.netlify.app/docs/latest/keyless/vault/readme/#part-2-configure-dns-for-the-oidc-discovery-ip-address

you will need to register a public DNS record that will resolve to the public IP address of your Kubernetes cluster.

#### Retrieve the IP Address of the SPIRE OIDC Discovery Provider
Run the following command to retrieve the external IP address of the spire-oidc service. The spire-oidc Discovery Provider service must provide an external IP address for Vault to access the OIDC Discovery document provided by spire-oidc.

$ kubectl get service -n spire spire-oidc

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE spire-oidc LoadBalancer 10.12.0.18 34.82.139.13 443:30198/TCP 108s ```

Create a hostname A record in https://my.noip.com/dynamic-dns , point it to public IP address of service in previous step

Verify the DNS A Record

$ nslookup oidc-discovery.example.org
Server:        203.0.113.0
Address:          203.0.113.0#53

Non-authoritative answer:
Name:   oidc-discovery.example.org
Address: 93.184.216.34

The Address: field at the bottom should correspond to the IP address in the A record.

In your browser, navigate to https://MY_DISCOVERY_DOMAIN/.well-known/openid-configuration. You should see JSON output similar to the following:

{
  "issuer": "https://oidc-discovery.example.org",
  "jwks_uri": "https://oidc-discovery.example.org/keys",
  "authorization_endpoint": "",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [],
  "id_token_signing_alg_values_supported": [
    "RS256",
    "ES256",
    "ES384"
  ]
}

References:

Azure Design tools

2021-04-04T12:03:38+01:00

In this article I like to give you an overview about resources that helps you to visualize and document your Azure cloud solutions and environments.

Shapes and icon sets

Azure Design Visio VSSX

David Summers started an "Azure Icon Design project" to collect icons in Visio format (VSSX). You'll find the icon set from his GitHub repo:
https://github.com/David-Summers/Azure-Design

Azure-Stencils (VSSX und SVG)

There is just another GitHub repository by "azurekid" with many (categorized) icons in Visio format (VSSX) and SVG:
https://github.com/azurekid/Azure-Stencils

Microsoft Azure UX Patterns

Few hundreds icons in SVG format are also available from the "official" Microsoft Azure site:
https://azure.microsoft.com/en-us/patterns/styles/glyphs-icons/

Microsoft Integration Stencils Pack for Visio

Microsoft updates regular this Visio stencils package with many symbols of on-premises, hybrid and cloud services/products: https://gallery.technet.microsoft.com/Collection-of-Integration-e6a3f4d0#content

Microsoft Azure, Cloud and Enterprise Symbol / Icon Set

The most updated collection of cloud related icons by Microsoft is the following one. It includes (categorized) symbols in SVG format: https://www.microsoft.com/en-us/download/details.aspx?id=41937

Drawing your architecture (Samples)

Azure Solutions Architectures

Microsoft's Azure solution architecture page shows some well-designed samples of reference architectures. This could be an inspiration to design your own architecture draws:
https://azure.microsoft.com/en-us/solutions/architecture/

Microsoft 3D Visio Templates

You have probably already seen some beautiful Azure 3D blueprints from Microsoft (in the recent years). In this video you will get an introduction of using a Visio template that allows to create such isometric blueprints:
Microsoft 3D Blueprint Visio Training Video

Microsoft has moved to a flatter style in the current architecture draws. The development of this template has stopped in 2017 but it's still available from the Microsoft Download-Center:
Microsoft 3D Blueprint Visio Template Download

Online Diagram Tools

Visual Paradigm Online Diagram

This cross-platform diagram solution includes standard Azure icons and samples to start drawing your own Azure diagram:
https://online.visual-paradigm.com

Draw.io

The following diagram solution is a very popular free-to-license web app with integration in Atlassian products and other features (e.g. VSSX import, OneDrive-support). Check out the various "Cloud" templates (including AWS, Azure and GCP) by clicking on "create new diagrams".
I can strongly recommended to give them a try.
https://about.draw.io/

Represent Azure architectures accurately in Visio Online

You are using Vision Online? Take a look on the library which includes already Azure symbols, templates, and sample diagrams: https://techcommunity.microsoft.com/t5/Visio-Blog/Represent-Azure-architectures-accurately-in-Visio-Online/ba-p/274650

Visualizer and Automated Documentation

Azure Resource Visualizer

Using ARM parameter and template files are parts of Azure's native "Infrastructure as Code" approach but are also one of the best ways to document your deployment. Visualizing your Azure resources based on your ARM files is (in my opinion) also a great option. Even in your draft phase this helps to visualize and review your planned solution.

That's why ARMViz was my first choice over the past years - available as online tool or install package (to run it in your own environment): https://github.com/ytechie/AzureResourceVisualizer http://armviz.io/designer

Unfortunately it seems that ARMViz is outdated and have been abandoned. Ben Coleman started a project ("ARM Template Viewer") and released a first version of a VSCode extension to displays a graphical view of an opened ARM template file. Great user experience and value for ARM authors! Strongly recommended to check out the extension: https://marketplace.visualstudio.com/items?itemName=bencoleman.armview

Cloudockit

This tool allows you to generate documentation and diagrams for your Azure and other cloud service platforms (AWS, GCP, ...) environment as well. Exportable as document type for Excel, Word, Visio and Draw.io.
Product details (subscription price) and trial version are available here:
https://www.cloudockit.com

CloudCraft.co

Easy to use and popular architecture diagram generator. Unfortunately (full) support only for AWS environment (status: 09/30/2019). Advanced collaboration and editing/export features (e.g. Draw.io support) included.
Free version and pricing details are available:
http://cloudcraft.co

General advice on writing design documentations

Define an efficient documentation structure: Start with a short introduction of your key points such as scope, business/compliance requirements or technical background (initial situation). All your design documentations should follow a common schema.
Describe the relation of your designed solution to the general (cloud) architecture: This is very important if your solution has dependency or strong references to other design decisions or guidelines.
Start with a picture instead of using thousand words: Describe your solution based on a well-designed architecture draw (in formats that can be easily modified by design changes)
Capture the decisions (including pros and cons): What were the reasons for the planned solutions? Review your design decision and the reasons to having cut off other variants?
Design vs. implementation/operation guide: Move details about implementation or specification to the appendix or splitting the documentation between architecture (design) document and implementation guide.

If you want to learn more about architecture and design patterns in Azure this study collection might be interesting for you.

Architecture & Patterns

Azure for Architects (Free Edition)

You are looking for a book to start with the foundation of designing solutions in Azure? General guidance of core topics e.g. Azure Resource Manager (Templates), architectural considerations and overview of some Azure services are included in the following book. You'll receive a free copy (e-book) from Microsoft:
https://azure.microsoft.com/en-us/resources/azure-for-architects/

AzureCAT Guidance

Microsoft's CAT stands for "Customer Advisory Team" and actively assists with customers in complex projects and take the learning backs to the product teams. You'll find the white papers and resources of CAT here:
http://aka.ms/CAT

Azure Architecture Center

This is one of the most valuable sources for cloud architects. Including reference architecture of the most common solutions in Azure. Microsoft's cloud adoption framework and design patterns are also part of this library.
https://docs.microsoft.com/en-us/azure/architecture/

Azure Security Architecture

"Shared responsibility" and "modern perimeter" ("Zero Trust") are just few aspects of cloud transformation and IT modernization in mostly every organization. This section of the "Azure Architecture center" describes many best practices, reference architectures and recommendations that are related for architects or security administrators.
https://docs.microsoft.com/en-us/azure/architecture/security/overview

Azure Application Architecture Guide

This guide gives you an overview and introduction on how to design and implement software architecture and applications in Azure. Great resource if you are interested to compare traditional on-premises and modernized cloud approaches.
https://docs.microsoft.com/azure/architecture/guide/

Microsoft 365 Enterprise foundation infrastructure

First you should build a strong foundation before you're starting the deployment of Microsoft 365 in your organization. The following deployment guide (divided up into several phases) is a great way for planning the most foundational subjects (from Network, Identity until Mobile Devices) on your journey to introduce Microsoft 365 services:
https://docs.microsoft.com/en-us/microsoft-365/enterprise/deploy-foundation-infrastructure

Microsoft Cloud IT architecture resources

If you like to learn more about core cloud architecture concepts for Microsoft identity, security and network this resource is very helpful:
https://docs.microsoft.com/en-us/office365/enterprise/microsoft-cloud-it-architecture-resources)

Serverless apps: Architecture, patterns and Azure implementations

The following e-book is a perfect (starter) guide to get an overview of "Serverless apps" in Azure. This includes benefits, potential solutions and different design patterns in developing cloud native development using serverless computing.
https://aka.ms/serverless-ebook

Cloud Computing Services

Enterprise Cloud Strategy

Definition of cloud strategy and roadmap are a key point before starting the transformation of an organization. You can get a free copy of the e-book "Enterprise Cloud Strategy" by Microsoft which includes general aspects of moving to cloud computing (costs, cloud models, etc.):
https://azure.microsoft.com/en-us/resources/enterprise-cloud-strategy/

Compare Cloud solutions

In a world of various cloud services and provides it is hard to compare and keep the different product names in mind. This page helps to compare the name of several offerings between Azure, AWS, IBM and other major providers:
http://comparecloud.in/

How I choose which services to use in Azure?

This is one of the most common question if you are designing or migrating services. The following articles shows a good approach to find the right answer:
* How I choose which services to use in Azure?
* Decision tree for Azure compute services
* Criteria for choosing an Azure compute service

Original content here: https://www.cloud-architekt.net/

Run IBM Datapower Gateway IDG-10.0.0 on Docker

2020-10-30T00:00:00+00:00

To build your IBM DataPower Docker image, you must develop the application's configuration. The easiest method is in the Docker containers on your workstation.

Before you begin

Before you can build and deploy a DataPower Docker application you must create an export package that contains the DataPower configuration for the DataPower Docker image. You create and export the DataPower configuration on a DataPower appliance or virtual DataPower offering.

You can create the DataPower configuration using the DataPower GUI, CLI, or other management interface, which can be importing an existing export package from a secure server and using a deployment policy with deployment policy variables to modify the configuration in the export package during import. The resultant and exported configuration should be the explicit configuration for your DataPower Docker application.

The defined and imported configuration is restricted to features supported by DataPower for Docker. If you create an export package from another DataPower offering with features not supported by DataPower Gateway for Docker, these feature will be unavailable. For more information, see Managing add-on features for a DataPower Docker image.

The easiest way to export and import packages is through the DataPower GUI, but you can use the DataPower backup command to do an export.

Procedure

Download the version-specific DataPower firmware image from the read-only IBM Entitled Registry.
Create a clean working directory with the config, local, and certs subdirectories. These subdirectories will be mounted inside the container to extract the application's configuration. For more information, see Directories in a DataPower Docker image.
Grant full permission to ensure that everyone can access these subdirectories.

chmod -R 777 config local certsCopy code

Start the container. The following snippet is the minimum required set of parameters. For more information, see Environment variables and drouter arguments and a DataPower Docker image.

docker run -it --name name \ -v $(pwd)/config:/opt/ibm/datapower/drouter/config \ -v $(pwd)/local:/opt/ibm/datapower/drouter/local \ -v $(pwd)/certs:/opt/ibm/datapower/root/secure/usrcerts \ -e DATAPOWER_ACCEPT_LICENSE="true" \ -e DATAPOWER_INTERACTIVE="true" \ -p 9090:9090 \ tag

Where name is the name of the container, and tag is generally in the registry-path:version.build-edition format.

Configure access to the DataPower GUI.

$$ configure terminal $$ web-mgmt $$ admin-state "enabled" $$ exit

Access the DataPower Gateway to import the export package that contains your DataPower configuration.
- To start a GUI session, enter https://localhost:9090 as the URL in your browser.
- To start a CLI session, use the docker attach command.

After you write and test your configuration, save everything to your mounted volumes.
In the GUI, click Save Configuration.
In the CLI, issue the write memory command.

Stop the DataPower container, where name is the name of the container.

docker stop -t 300 name

Change ownership of files owned by root.

chown -R $USER:$USER config local certs

Create the Dockerfile for the DataPower Docker image. The following snippet is the most basic Dockerfile that you should require.

FROM tag COPY config /opt/ibm/datapower/drouter/config COPY local /opt/ibm/datapower/drouter/local COPY certs /opt/ibm/datapower/root/secure/usrcerts USER root RUN chown -R drouter:drouter /opt/ibm/datapower/drouter/config \ /opt/ibm/datapower/drouter/local \ /opt/ibm/datapower/root/secure/usrcerts RUN set-user drouter USER drouter

With your Dockerfile, build your DataPower Docker image, where my-image is the name that differentiates various DataPower Docker images in your repository.

docker build . -f Dockerfile -t my-imageCopy code

Use the docker push command to upload the DataPower Docker image to your repository.

References:

https://docs.docker.com/docker-for-windows/
https://linuxconfig.org/docker-container-backup-and-recovery
https://hub.docker.com/r/ibmcom/datapower/
https://logixbubble.wordpress.com/2017/10/30/virtual-datapower-setup/
https://docs.docker.com/engine/reference/commandline/port/
https://hub.docker.com/r/middlewareaman/datapower_av
https://www.ibm.com/support/knowledgecenter/en/SS9H2Y_7.7.0/com.ibm.dp.doc/request-header_metadatafunction.html
http://index-of.co.uk/Tutorials/XML%20for%20Dummies%204th%20Ed.pdf
DataPower MPGW Simple Exercise - https://www.youtube.com/watch?v=mz7VBIbyjnI
https://www.ibm.com/support/knowledgecenter/en/SS9H2Y_10.0/com.ibm.dp.doc/welcome.html
https://github.com/amanverma-18/datapower_publiccerts
https://github.com/dan-orangespecs/practical-datapower http://www.orangespecs.com/category/datapower/
https://www.ibm.com/support/pages/migration-ssl-proxy-profile
https://www.youtube.com/watch?v=i0xaZ-iaEbU&list=PLw_7CPUqcdrFMnBmRVQENC_AlcaCCmVtd&index=3
https://www.ibm.com/support/knowledgecenter/SS9H2Y_10.0/com.ibm.dp.doc/apigw_overview.html
https://integrationtechies.wordpress.com/topics/
https://www.ibm.com/support/knowledgecenter/SS9H2Y_10.0/com.ibm.dp.doc/docker_dpapp.html

Commands:

docker run -it --name dpcontainer02 -v /e/development_work/datapower/datapowerconfig:/drouter/config -v /e/development_work/datapower/datapowerlocal:/drouter/local -e DATAPOWER_ACCEPT_LICENSE=true -e DATAPOWER_INTERACTIVE=true -e DATAPOWER_WORKER_THREADS=4 -p 9090:9090 -p 9022:22 -p 5554:5554 -p 8000-8100:8000-8100 middlewareaman/datapower_av:version10.0

Convert Swagger or OpenAPI Specs to Spring Boot REST API

2020-08-06T00:00:00+01:00

What is Swagger? Swagger allows you to describe the structure of your APIs so that machines can read them. The ability of APIs to describe their own structure is the root of all awesomeness in Swagger.

Open https://editor.swagger.io/
Select File -> Import URL
Enter any of these URL and you would be able to see compelete spec in a clearly understandable format.
API ServiceOrdering 4.0.0 [ Base URL: serverRoot/tmf-api/serviceOrdering/v4 ]
Federated ID 4.0 [ Base URL: serverRoot/tmf-api/openid/v4 ]
Communication Management API 4.0.0 [ Base URL: serverRoot/tmf-api/communicationManagement/v4/ ]

You would notice how OpenAPI/Swagger Specification helps understand the API and agree on its attributes. In this post I am going to show how we can generate code and documentation from the specification file.

Let's generate the code create an empty maven project named "TMF-ApiSpec"

There are several ways to generate the code from a swagger json/yaml spec.
For e.g. You can download openapi-generator-cli-4.2.3.jar and use the generate command.

java -cp . -jar openapi-generator-cli-4.2.3.jar generate -i TMF641-ServiceOrdering-3.0.0.swagger.json --api-package com.cts.serviceorder.client.api --model-package com.cts.serviceorder.client.model --invoker-package com.cts.serviceorder.client.invoker --group-id com.cts --artifact-id spring-openapi-generator-api-client --artifact-version 0.0.1-SNAPSHOT -g java -p java8=true --library resttemplate -o openapi-serviceorder

See more about generate and help

You can also use online service to generate the code:

curl -X POST -H "content-type:application/json" -d '{"openAPIUrl":"https://github.com/tmforum-apis/TMF641_ServiceOrder/releases/download/v3.0.0/TMF641-ServiceOrdering-3.0.0.swagger.json"}' http://api.openapi-generator.tech/api/gen/clients/java

I am using the plugin

For the API spec I am using TMForum's ServiceOrderingManagement OpenAPI specification. You can choose one of Yaml or Json format

Update the pom.xml File:

  4.0.0
  com.ashwani
  TMF-ApiSpec
  0.0.1-SNAPSHOT
 
        1.5.22
        2.27
        2.8.9
        2.7
        1.0.0
        4.8.1
        2.9.2
        1.3.8
        2.6.4
        2.1.1.RELEASE
        2.1.0.RELEASE
        4.12
        2.2
    
            io.swagger
            swagger-annotations
            ${swagger-annotations-version}
        
            org.glassfish.jersey.core
            jersey-client
            ${jersey-version}
        
            org.glassfish.jersey.media
            jersey-media-multipart
            ${jersey-version}
        
            org.glassfish.jersey.media
            jersey-media-json-jackson
            ${jersey-version}
        
            com.fasterxml.jackson.jaxrs
            jackson-jaxrs-base
            ${jackson-version}
        
            com.fasterxml.jackson.core
            jackson-core
            ${jackson-version}
        
            com.fasterxml.jackson.core
            jackson-annotations
            ${jackson-version}
        
            com.fasterxml.jackson.core
            jackson-databind
            ${jackson-version}
        
            com.fasterxml.jackson.jaxrs
            jackson-jaxrs-json-provider
            ${jackson-version}
        
            com.fasterxml.jackson.datatype
            jackson-datatype-joda
            ${jackson-version}
        
            joda-time
            joda-time
            ${jodatime-version}
        
            com.brsanthu
            migbase64
            ${migbase64-version}
        
            junit
            junit
            ${junit-version}
            test
        
            org.springframework.boot
            spring-boot-starter-test
            ${spring-boot-starter-test-version}
            test
        
            org.springframework.boot
            spring-boot-starter-web
            ${spring-boot-starter-web-version}
        
            io.springfox
            springfox-swagger2
            ${springfox-version}
        
            io.springfox
            springfox-swagger-ui
            ${springfox-version}
        
            org.threeten
            threetenbp
            ${threetenbp-version}
        
            com.github.joschi.jackson
            jackson-datatype-threetenbp
            ${datatype-threetenbp-version}
        
                org.openapitools
                openapi-generator-maven-plugin
                3.3.4
                
                        spring-boot-api
                        
                            generate
                        
                            ${project.basedir}/src/main/resources/TMF641-Service_Ordering-v4.0.0.swagger.json
                            spring
                            
                                joda
                            
                            spring-boot
                            com.ashwani.demo.api
                            com.ashwani.demo.api.model
                            com.ashwani.demo.api.handler
                        
                org.apache.maven.plugins
                maven-compiler-plugin
                3.6.1
                
                    1.8
                    1.8
                
                maven-deploy-plugin
                2.8.1
                
                        default-deploy
                        deploy
                        
                            deploy

Execute MVN install in the root directory of the project. And the resulting code would be generated by openapi-generator-maven-plugin in target/generated-sources/. com.ashwani.demo.api. Various generators based on your preference.

Now let's create a new spring boot project demo-service from https://start.spring.io/. Once a bare spring boot project is created with Spring Web dependency. Now add the Maven dependency in this spring project.

              com.ashwani
              openapi-serviceorder-client
          0.0.1-SNAPSHOT
          system
          ${basedir}/lib/TMF-ApiSpec-0.0.1-SNAPSHOT.jar
      

With the other dependecies for this project , the pom.xml should look like this:

  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  4.0.0
  
      org.springframework.boot
      spring-boot-starter-parent
      2.3.2.RELEASE
       
  com.ashwani
  TMF-OpenAPISim
  0.0.1-SNAPSHOT
  TMF-OpenAPISim
  Spring Boot - TMF-OpenAPISim

        9
        ${java.version}
        ${java.version}
        2.8.0
    
          org.springframework.boot
          spring-boot-starter-web
      
            io.springfox
            springfox-swagger2
            ${springfox-version}
        
          javax.xml.bind
          jaxb-api
          2.3.0
      
            io.springfox
            springfox-swagger-ui
            ${springfox-version}
        
            com.fasterxml.jackson.datatype
            jackson-datatype-joda
        
            javax.validation
            validation-api
        
                com.github.joschi.jackson
                jackson-datatype-threetenbp
                2.9.10
            
          org.springframework.boot
          spring-boot-starter-test
          test
          
                  org.junit.vintage
                  junit-vintage-engine
              
          com.ashwani
          openapi-serviceorder-client
          0.0.1-SNAPSHOT
          system
          ${basedir}/lib/TMF-ApiSpec-0.0.1-SNAPSHOT.jar
      
            io.springfox
            springfox-swagger-ui
            ${springfox-version}
        
              org.springframework.boot
              spring-boot-maven-plugin

Now create a class ServiceOrderApiController which will implement previously generated ServiceOrderApi from previous step. And make some implementation of the API:

package com.ashwani.serviceorder.api;

import java.security.Principal;
import java.util.Optional;

import javax.servlet.http.HttpServletRequest;
import javax.validation.Valid;

import org.joda.time.DateTime;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.threeten.bp.OffsetDateTime;

import com.ashwani.serviceorder.api.model.Error;
import com.ashwani.serviceorder.api.model.RelatedParty;
import com.ashwani.serviceorder.api.model.ServiceOrder;
import com.ashwani.serviceorder.api.model.ServiceOrderCreate;
import com.ashwani.serviceorder.api.model.ServiceOrderItem;
import com.ashwani.serviceorder.api.model.ServiceRestriction;
import com.ashwani.serviceorder.api.ServiceOrderRepoService;


import com.fasterxml.jackson.databind.ObjectMapper;

import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.ApiParam;
import io.swagger.annotations.ApiResponse;
import io.swagger.annotations.ApiResponses;

@javax.annotation.Generated(value = "org.openapitools.codegen.languages.SpringCodegen", date = "2020-07-25T00:30:31.539259300+01:00[Europe/London]")

@Controller
@RequestMapping("${openapi.aPIServiceOrdering.base-path:/tmf-api/serviceOrdering/v3}")
public class ServiceOrderApiController implements ServiceOrderApi {

  private static final Logger logger = LoggerFactory.getLogger(ServiceOrderApiController.class);

  private final ObjectMapper objectMapper;

  private final HttpServletRequest request;
  
  @Autowired
  ServiceOrderRepoService serviceOrderRepoService;

  
    @Autowired
  public ServiceOrderApiController(ObjectMapper objectMapper, HttpServletRequest request) {
      this.objectMapper = objectMapper;
      this.request = request;
  }

    @ApiOperation(value = "Creates a ServiceOrder", nickname = "createServiceOrder", notes = "This operation creates a ServiceOrder entity.", response = ServiceOrder.class, tags={ "serviceOrder", })
    @ApiResponses(value = { 
        @ApiResponse(code = 201, message = "Created", response = ServiceOrder.class),
        @ApiResponse(code = 400, message = "Bad Request", response = Error.class),
        @ApiResponse(code = 401, message = "Unauthorized", response = Error.class),
        @ApiResponse(code = 403, message = "Forbidden", response = Error.class),
        @ApiResponse(code = 405, message = "Method Not allowed", response = Error.class),
        @ApiResponse(code = 409, message = "Conflict", response = Error.class),
        @ApiResponse(code = 500, message = "Internal Server Error", response = Error.class) })
    @RequestMapping(value = "/serviceOrder",
        produces = { "application/json;charset=utf-8" }, 
        consumes = { "application/json;charset=utf-8" },
        method = RequestMethod.POST)
    public ResponseEntity createServiceOrder(@ApiParam(value = "The ServiceOrder to be created" ,required=true )  @Valid @RequestBody ServiceOrderCreate serviceOrderCreate) {
        ServiceOrderCreate soc = serviceOrderCreate;
        
            try {
                  System.out.println(serviceOrderCreate.toString());
                  ServiceOrder c = serviceOrderRepoService.addServiceOrder(soc);
                  return new ResponseEntity(c, HttpStatus.OK);               

          }catch (Exception e) {
              e.printStackTrace();
              return new ResponseEntity(HttpStatus.INTERNAL_SERVER_ERROR);
          }
    }
}

package com.ashwani.serviceorder.api;


import java.util.Date;
import java.util.Set;

import javax.validation.Valid;

import org.joda.time.DateTime;
import org.springframework.stereotype.Service;
import org.threeten.bp.OffsetDateTime;
import org.threeten.bp.ZoneOffset;

import com.ashwani.serviceorder.api.model.Any;
import com.ashwani.serviceorder.api.model.Characteristic;
import com.ashwani.serviceorder.api.model.ServiceOrder;
import com.ashwani.serviceorder.api.model.ServiceOrderAttributeValueChangeEvent;
import com.ashwani.serviceorder.api.model.ServiceOrderAttributeValueChangeNotification;
import com.ashwani.serviceorder.api.model.ServiceOrderCreate;
import com.ashwani.serviceorder.api.model.ServiceOrderItem;
import com.ashwani.serviceorder.api.model.ServiceOrderStateChangeEvent;
import com.ashwani.serviceorder.api.model.ServiceOrderStateChangeNotification;
import com.ashwani.serviceorder.api.model.ServiceOrderStateType;

@Service
public class ServiceOrderRepoService {

  public ServiceOrder addServiceOrder(@Valid ServiceOrderCreate serviceOrderCreate) {
      ServiceOrder so = new ServiceOrder();
      DateTime d = DateTime.now();
      so.setOrderDate(d);
      so.setCategory(serviceOrderCreate.getCategory());
      so.setDescription(serviceOrderCreate.getDescription());
      so.setExternalId(serviceOrderCreate.getExternalId());
      so.setNotificationContact(serviceOrderCreate.getNotificationContact());
      so.priority(serviceOrderCreate.getPriority());
      so.requestedCompletionDate(serviceOrderCreate.getRequestedCompletionDate());
      so.requestedStartDate(serviceOrderCreate.getRequestedStartDate());
      so.setExpectedCompletionDate(serviceOrderCreate.getRequestedCompletionDate()); // this is by default
      if (serviceOrderCreate.getNote() != null) {
          //so.getNote().addAll(serviceOrderCreate.getNote()); // --> This works with v3 of TMF Specification
          so.setNote(serviceOrderCreate.getNote());
      }

      boolean allAcknowledged = true;
      //if (serviceOrderCreate.getOrderItem() != null) { //--> This works with v3 of TMF Specification
          //so.getOrderItem().addAll(serviceOrderCreate.getOrderItem()); // --> This works with v3 of TMF Specification
          //for (ServiceOrderItem soi : so.getOrderItem()) {


      if (serviceOrderCreate.getServiceOrderItem() != null) {
          so.setServiceOrderItem(serviceOrderCreate.getServiceOrderItem());
          for (ServiceOrderItem soi : so.getServiceOrderItem()) { 
              //copySpecCharacteristicsToServiceCharacteristic(soi.getService().getServiceSpecification().getId(),
              //      soi.getService().getServiceCharacteristic());
              
              System.out.println("SOI --> " + soi.toString());
              
              if (soi.getState()!= null && !soi.getState().equals(ServiceOrderStateType.ACKNOWLEDGED)) {
                  allAcknowledged = false;
              }
          }

      }

      if (serviceOrderCreate.getRelatedParty() != null) {
          //so.getRelatedParty().addAll(serviceOrderCreate.getRelatedParty()); // --> This works with v3 of TMF Specification
          so.setRelatedParty(serviceOrderCreate.getRelatedParty());
      }
      if (serviceOrderCreate.getOrderRelationship() != null) {
          //so.getOrderRelationship().addAll(serviceOrderCreate.getOrderRelationship()); // --> This works with v3 of TMF Specification
          so.setOrderRelationship(serviceOrderCreate.getOrderRelationship());
      }


      // so = this.serviceOrderRepo.save(so);  You can save this in DB.

      if (allAcknowledged) { // in the case were order items are automatically acknowledged
          so.setState(ServiceOrderStateType.ACKNOWLEDGED);
          d = DateTime.now();
          so.setOrderDate(d);
          //so = this.serviceOrderRepo.save(so);
      }

      //raiseSOCreateNotification(so);

      return so;
  }
  
}

Now we can test the server. Send a request from postman on following end point. http://localhost:8080/tmf-api/serviceOrdering/v3/serviceOrder

Request Body

{
  "cancellationDate": "2020-08-08T11:55:14.450Z",
  "cancellationReason": "string",
  "category": "string",
  "description": "string",
  "externalId": "string",
  "notificationContact": "string",
  "priority": "string",
  "requestedCompletionDate": "2020-08-08T11:55:14.450Z",
  "requestedStartDate": "2020-08-08T11:55:14.450Z",
  "externalReference": [
    {
      "externalReferenceType": "string",
      "name": "string",
      "@baseType": "string",
      "@schemaLocation": "string",
      "@type": "string"
    }
  ],
  "note": [
    {
      "id": "string",
      "author": "string",
      "date": "2020-08-08T11:55:14.450Z",
      "text": "string",
      "@baseType": "string",
      "@schemaLocation": "string",
      "@type": "string"
    }
  ],
  "orderRelationship": [
    {
      "id": "string",
      "href": "string",
      "relationshipType": "string",
      "@baseType": "string",
      "@schemaLocation": "string",
      "@type": "string",
      "@referredType": "string"
    }
  ],
  "relatedParty": [
    {
      "id": "string",
      "href": "string",
      "name": "string",
      "role": "string",
      "@baseType": "ResourceSpecification",
      "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
      "@type": "LogicalResourceSpecification",
      "@referredType": "string"
    }
  ],
  "serviceOrderItem": [
    {
      "id": "string",
      "quantity": 0,
      "action": "add",
      "appointment": {
        "id": "string",
        "href": "string",
        "description": "string",
        "@baseType": "string",
        "@schemaLocation": "string",
        "@type": "string",
        "@referredType": "string"
      },
      "service": {
        "id": "string",
        "href": "string",
        "category": "string",
        "description": "string",
        "endDate": "2020-08-08T11:55:14.450Z",
        "hasStarted": true,
        "isBundle": true,
        "isServiceEnabled": true,
        "isStateful": true,
        "name": "string",
        "serviceDate": "string",
        "serviceType": "string",
        "startDate": "2020-08-08T11:55:14.450Z",
        "startMode": "string",
        "feature": [
          {
            "id": "string",
            "isBundle": true,
            "isEnabled": true,
            "name": "string",
            "constraint": [
              {
                "id": "string",
                "href": "string",
                "name": "string",
                "version": "string",
                "@baseType": "string",
                "@schemaLocation": "string",
                "@type": "string",
                "@referredType": "string"
              }
            ],
            "featureCharacteristic": [
              {
                "id": "string",
                "name": "string",
                "valueType": "string",
                "characteristicRelationship": [
                  {
                    "id": "string",
                    "relationshipType": "string",
                    "@baseType": "string",
                    "@schemaLocation": "string",
                    "@type": "string"
                  }
                ],
                "@baseType": "ResourceSpecification",
                "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                "@type": "LogicalResourceSpecification"
              }
            ],
            "featureRelationship": [
              {
                "id": "string",
                "name": "string",
                "relationshipType": "string",
                "validFor": {
                  "endDateTime": "1985-04-12T23:20:50.52Z",
                  "startDateTime": "1985-04-12T23:20:50.52Z",
                  "@baseType": "string",
                  "@schemaLocation": "string",
                  "@type": "string"
                },
                "@baseType": "string",
                "@schemaLocation": "string",
                "@type": "string"
              }
            ],
            "@baseType": "string",
            "@schemaLocation": "string",
            "@type": "string"
          }
        ],
        "note": [
          {
            "id": "string",
            "author": "string",
            "date": "2020-08-08T11:55:14.450Z",
            "text": "string",
            "@baseType": "string",
            "@schemaLocation": "string",
            "@type": "string"
          }
        ],
        "place": [
          {
            "id": "string",
            "href": "string",
            "name": "string",
            "role": "string",
            "@baseType": "ResourceSpecification",
            "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
            "@type": "LogicalResourceSpecification",
            "@referredType": "string"
          }
        ],
        "relatedEntity": [
          {
            "id": "string",
            "href": "string",
            "name": "string",
            "role": "string",
            "@baseType": "ResourceSpecification",
            "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
            "@type": "LogicalResourceSpecification",
            "@referredType": "string"
          }
        ],
        "relatedParty": [
          {
            "id": "string",
            "href": "string",
            "name": "string",
            "role": "string",
            "@baseType": "ResourceSpecification",
            "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
            "@type": "LogicalResourceSpecification",
            "@referredType": "string"
          }
        ],
        "serviceCharacteristic": [
          {
            "id": "string",
            "name": "string",
            "valueType": "string",
            "characteristicRelationship": [
              {
                "id": "string",
                "relationshipType": "string",
                "@baseType": "string",
                "@schemaLocation": "string",
                "@type": "string"
              }
            ],
            "@baseType": "ResourceSpecification",
            "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
            "@type": "LogicalResourceSpecification"
          }
        ],
        "serviceOrderItem": [
          {
            "itemId": "string",
            "role": "string",
            "serviceOrderHref": "string",
            "serviceOrderId": "string",
            "itemAction": "add",
            "@baseType": "string",
            "@schemaLocation": "string",
            "@type": "string",
            "@referredType": "string"
          }
        ],
        "serviceRelationship": [
          {
            "relationshipType": "string",
            "serviceRelationshipCharacteristic": [
              {
                "id": "string",
                "name": "string",
                "valueType": "string",
                "characteristicRelationship": [
                  {
                    "id": "string",
                    "relationshipType": "string",
                    "@baseType": "string",
                    "@schemaLocation": "string",
                    "@type": "string"
                  }
                ],
                "@baseType": "ResourceSpecification",
                "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                "@type": "LogicalResourceSpecification"
              }
            ],
            "@baseType": "string",
            "@schemaLocation": "string",
            "@type": "string"
          }
        ],
        "serviceSpecification": {
          "id": "string",
          "href": "string",
          "name": "string",
          "version": "string",
          "@baseType": "ResourceSpecification",
          "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
          "@type": "LogicalResourceSpecification",
          "@referredType": "string"
        },
        "state": "feasibilityChecked",
        "supportingResource": [
          {
            "id": "string",
            "href": "string",
            "name": "string",
            "@baseType": "ResourceSpecification",
            "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
            "@type": "LogicalResourceSpecification",
            "@referredType": "string"
          }
        ],
        "supportingService": [
          null
        ],
        "@baseType": "ResourceSpecification",
        "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
        "@type": "LogicalResourceSpecification",
        "@referredType": "string"
      },
      "serviceOrderItem": [
        null
      ],
      "serviceOrderItemRelationship": [
        {
          "relationshipType": "string",
          "orderItem": {
            "itemId": "string",
            "serviceOrderHref": "string",
            "serviceOrderId": "string",
            "@baseType": "ResourceSpecification",
            "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
            "@type": "LogicalResourceSpecification",
            "@referredType": "string"
          },
          "@baseType": "ResourceSpecification",
          "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
          "@type": "LogicalResourceSpecification"
        }
      ],
      "state": "acknowledged",
      "@baseType": "ResourceSpecification",
      "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
      "@type": "LogicalResourceSpecification"
    }
  ],
  "@baseType": "string",
  "@schemaLocation": "string",
  "@type": "string"
}

If all goes well you will get the following response:

Response Body

{
    "id": null,
    "href": null,
    "cancellationDate": null,
    "cancellationReason": null,
    "category": "string",
    "completionDate": null,
    "description": "string",
    "expectedCompletionDate": "2020-08-08T11:55:14.450Z",
    "externalId": "string",
    "notificationContact": "string",
    "orderDate": "2020-08-08T12:16:55.457Z",
    "priority": "string",
    "requestedCompletionDate": "2020-08-08T11:55:14.450Z",
    "requestedStartDate": "2020-08-08T11:55:14.450Z",
    "startDate": null,
    "externalReference": null,
    "note": [
        {
            "id": "string",
            "author": "string",
            "date": "2020-08-08T11:55:14.450Z",
            "text": "string",
            "@baseType": "string",
            "@schemaLocation": "string",
            "@type": "string"
        }
    ],
    "orderRelationship": [
        {
            "id": "string",
            "href": "string",
            "relationshipType": "string",
            "@baseType": "string",
            "@schemaLocation": "string",
            "@type": "string",
            "@referredType": "string"
        }
    ],
    "relatedParty": [
        {
            "id": "string",
            "href": "string",
            "name": "string",
            "role": "string",
            "@baseType": "ResourceSpecification",
            "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
            "@type": "LogicalResourceSpecification",
            "@referredType": "string"
        }
    ],
    "serviceOrderItem": [
        {
            "id": "string",
            "quantity": 0,
            "action": "add",
            "appointment": {
                "id": "string",
                "href": "string",
                "description": "string",
                "@baseType": "string",
                "@schemaLocation": "string",
                "@type": "string",
                "@referredType": "string"
            },
            "service": {
                "id": "string",
                "href": "string",
                "category": "string",
                "description": "string",
                "endDate": "2020-08-08T11:55:14.450Z",
                "hasStarted": true,
                "isBundle": true,
                "isServiceEnabled": true,
                "isStateful": true,
                "name": "string",
                "serviceDate": "string",
                "serviceType": "string",
                "startDate": "2020-08-08T11:55:14.450Z",
                "startMode": "string",
                "feature": [
                    {
                        "id": "string",
                        "isBundle": true,
                        "isEnabled": true,
                        "name": "string",
                        "constraint": [
                            {
                                "id": "string",
                                "href": "string",
                                "name": "string",
                                "version": "string",
                                "@baseType": "string",
                                "@schemaLocation": "string",
                                "@type": "string",
                                "@referredType": "string"
                            }
                        ],
                        "featureCharacteristic": [
                            {
                                "id": "string",
                                "name": "string",
                                "valueType": "string",
                                "characteristicRelationship": [
                                    {
                                        "id": "string",
                                        "relationshipType": "string",
                                        "@baseType": "string",
                                        "@schemaLocation": "string",
                                        "@type": "string"
                                    }
                                ],
                                "value": null,
                                "@baseType": "ResourceSpecification",
                                "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                                "@type": "LogicalResourceSpecification"
                            }
                        ],
                        "featureRelationship": [
                            {
                                "id": "string",
                                "name": "string",
                                "relationshipType": "string",
                                "validFor": {
                                    "endDateTime": "1985-04-12T23:20:50.520Z",
                                    "startDateTime": "1985-04-12T23:20:50.520Z",
                                    "@baseType": "string",
                                    "@schemaLocation": "string",
                                    "@type": "string"
                                },
                                "@baseType": "string",
                                "@schemaLocation": "string",
                                "@type": "string"
                            }
                        ],
                        "@baseType": "string",
                        "@schemaLocation": "string",
                        "@type": "string"
                    }
                ],
                "note": [
                    {
                        "id": "string",
                        "author": "string",
                        "date": "2020-08-08T11:55:14.450Z",
                        "text": "string",
                        "@baseType": "string",
                        "@schemaLocation": "string",
                        "@type": "string"
                    }
                ],
                "place": [
                    {
                        "id": "string",
                        "href": "string",
                        "name": "string",
                        "role": "string",
                        "@baseType": "ResourceSpecification",
                        "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                        "@type": "LogicalResourceSpecification",
                        "@referredType": "string"
                    }
                ],
                "relatedEntity": [
                    {
                        "id": "string",
                        "href": "string",
                        "name": "string",
                        "role": "string",
                        "@baseType": "ResourceSpecification",
                        "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                        "@type": "LogicalResourceSpecification",
                        "@referredType": "string"
                    }
                ],
                "relatedParty": [
                    {
                        "id": "string",
                        "href": "string",
                        "name": "string",
                        "role": "string",
                        "@baseType": "ResourceSpecification",
                        "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                        "@type": "LogicalResourceSpecification",
                        "@referredType": "string"
                    }
                ],
                "serviceCharacteristic": [
                    {
                        "id": "string",
                        "name": "string",
                        "valueType": "string",
                        "characteristicRelationship": [
                            {
                                "id": "string",
                                "relationshipType": "string",
                                "@baseType": "string",
                                "@schemaLocation": "string",
                                "@type": "string"
                            }
                        ],
                        "value": null,
                        "@baseType": "ResourceSpecification",
                        "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                        "@type": "LogicalResourceSpecification"
                    }
                ],
                "serviceOrderItem": [
                    {
                        "itemId": "string",
                        "role": "string",
                        "serviceOrderHref": "string",
                        "serviceOrderId": "string",
                        "itemAction": "add",
                        "@baseType": "string",
                        "@schemaLocation": "string",
                        "@type": "string",
                        "@referredType": "string"
                    }
                ],
                "serviceRelationship": [
                    {
                        "relationshipType": "string",
                        "service": null,
                        "serviceRelationshipCharacteristic": [
                            {
                                "id": "string",
                                "name": "string",
                                "valueType": "string",
                                "characteristicRelationship": [
                                    {
                                        "id": "string",
                                        "relationshipType": "string",
                                        "@baseType": "string",
                                        "@schemaLocation": "string",
                                        "@type": "string"
                                    }
                                ],
                                "value": null,
                                "@baseType": "ResourceSpecification",
                                "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                                "@type": "LogicalResourceSpecification"
                            }
                        ],
                        "@baseType": "string",
                        "@schemaLocation": "string",
                        "@type": "string"
                    }
                ],
                "serviceSpecification": {
                    "id": "string",
                    "href": "string",
                    "name": "string",
                    "version": "string",
                    "@baseType": "ResourceSpecification",
                    "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                    "@type": "LogicalResourceSpecification",
                    "@referredType": "string"
                },
                "state": "feasibilityChecked",
                "supportingResource": [
                    {
                        "id": "string",
                        "href": "string",
                        "name": "string",
                        "@baseType": "ResourceSpecification",
                        "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                        "@type": "LogicalResourceSpecification",
                        "@referredType": "string"
                    }
                ],
                "supportingService": [
                    null
                ],
                "@baseType": "ResourceSpecification",
                "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                "@type": "LogicalResourceSpecification",
                "@referredType": "string"
            },
            "serviceOrderItem": [
                null
            ],
            "serviceOrderItemRelationship": [
                {
                    "relationshipType": "string",
                    "orderItem": {
                        "itemId": "string",
                        "serviceOrderHref": "string",
                        "serviceOrderId": "string",
                        "@baseType": "ResourceSpecification",
                        "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                        "@type": "LogicalResourceSpecification",
                        "@referredType": "string"
                    },
                    "@baseType": "ResourceSpecification",
                    "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
                    "@type": "LogicalResourceSpecification"
                }
            ],
            "state": "acknowledged",
            "@baseType": "ResourceSpecification",
            "@schemaLocation": "https://mycsp.com:8080/tmf-api/schema/Resource/LogicalResourceSpecification.schema.json",
            "@type": "LogicalResourceSpecification"
        }
    ],
    "state": null,
    "@baseType": null,
    "@schemaLocation": null,
    "@type": null
}

Next steps:

To add persistent to the order creation etc. Something similar to this

Few References:

Swagger-codegen/ Swagger Editor online TMForum OpenAPI Table openapi-generator-maven-plugin Swagger UI-Once you generate the code, it also generates a swagger UI openslice TMForum Datamodel

Python Vs Java - Uses, Performance, Learning

2020-07-09T00:00:00+01:00

Python vs. Java: Uses, Performance, Learning

In the world of computer science, there are many programming languages, and no single language is superior to another.
In other words, each language is best suited to solve certain problems, and in fact there is often no one best language to choose for a given programming project.
For this reason, it is important for students who wish to develop software or to solve interesting problems through code to have strong computer science fundamentals that will apply across any programming language.

Programming languages tend to share certain characteristics in how they function, for example in the way they deal with memory usage or how heavily they use objects.
Students will start seeing these patterns as they are exposed to more languages. This article will focus primarily on Python versus Java, which are two of the most widely used programming languages in the world. While it is hard to measure exactly the rate at which each programming language is growing, these are two of the most popular programming languages used in industry today. One major difference between Python and Java is that Python is dynamically typed, while Java is statically typed. Loosely, this means that Java is much more strict about how variables are defined and used in code.
As a result, Java tends to be more verbose in its syntax, which is one of the reasons we recommend learning Python before Java for beginners.
For example, here is how you would create a variable named numbers that holds the numbers 0 through 9 in Python:

numbers = [] for i in range(10): numbers.append(i)

Here's how you would do the same thing in Java: ArrayList numbers = new ArrayList();

for (int i = 0; i < 10; i++) { numbers.add(i); }

Another major difference is that Java generally runs programs more quickly than Python, as it is a compiled language.
This means that before a program is actually run, the compiler translates the Java code into machine-level code.
By contrast, Python is an interpreted language, meaning there is no compile step.

Usage and Practicality

Historically, Java has been the more popular language in part due to its lengthy legacy. However, Python is rapidly gaining ground. According to Github.s State of the Octoberst Report, it has recently surpassed Java as the most widely used programming language. As per the 2018 developer survey, Python is now the fastest-growing computer programing language. Both Python and Java have large communities of developers to answer questions on websites like Stack Overflow. As you can see from Stack Overflow trends, Python surpassed Java in terms the percentage of questions asked about it on Stack Overflow in 2017. At the time of writing, about 13% of the questions on Stack Overflow are tagged with Python, while about 8% are tagged with Java!

Web Development

Python and Java can both be used for backend web development. Typically developers will use the Django and Flask frameworks for Python and Spring for Java. Python is known for its code readability, meaning Python code is clean, readable, and concise. Python also has a large, comprehensive set of modules, packages, and libraries that exist beyond its standard library, developed by the community of Python enthusiasts. Java has a similar ecosystem, although perhaps to a lesser extent.

Mobile App Development

In terms of mobile app development, Java dominates the field, as it is the primary langauge used for building Android apps and games. Thanks to the aforementioned tailored libraries, developers have the option to write Android apps by leveraging robust frameworks and development tools built specifically for the operating system. Currently, Python is not used commonly for mobile development, although there are tools like Kivy and BeeWare that allow you to write code once and deploy apps across Windows, OS X, iOS, and Android.

Machine Learning and Big Data

Conversely, in the world of machine learning and data science, Python is the most popular language. Python is often used for big data, scientific computing, and artificial intelligence (A.I.) projects. The vast majority of data scientists and machine learning programmers opt for Python over Java while working on projects that involve sentiment analysis. At the same time, it is important to note that many machine learning programmers may choose to use Java while they work on projects related to network security, cyber attack prevention, and fraud detection.

Where to Start

When it comes to learning the foundations of programming, many studies have concluded that it is easier to learn Python over Java, due to Python's simple and intuitive syntax, as seen in the earlier example. Java programs often have more boilerplate code - sections of code that have to be included in many places with little or no alteration - than Python. That being said, there are some notable advantages to Java, in particular its speed as a compiled language. Learning both Python and Java will give students exposure to two languages that lay their foundation on similar computer science concepts, yet differ in educational ways. Overall, it is clear that both Python and Java are powerful programming languages in practice, and it would be advisable for any aspiring software developer to learn both languages proficiently. Programmers should compare Python and Java based on the specific needs of each software development project, as opposed to simply learning the one language that they prefer. In short, neither language is superior to another, and programmers should aim to have both in their coding experience.

Score card

Runtime Performance (Winner - Java)
Ease of Learning (Winner - Python)
Practical Agility (Tie b/w Java and Python)
Mobile App Development (Winner - Java)
Big Data (Winner - Python)

Disclaimer: I recenty came across a startup junilearning.com - This Startup Connects Your Kid With Ivy League Student Instructors. You should go check it out.

This is Collaboration post with junilearning.com and this post is directly published on collaboration request from Jasmine Gordon jasmine.x.xxxxxx@junilearning.co This article originally appeared on junilearning.com

Symbols Used in Architecture Diagrams

2020-07-03T00:00:00+01:00

Sometime back I was looking for these standard #symbols to demonstrate level of change for each system and interface.
The key to the symbols used in attached to each of the overall #technical #architecture #diagrams and a full definition is given in this article.

Some of the following symbols are used in the architecture diagrams (there can be more).

The Architecture diagrams show the level of change for each system and interface. The key to the symbols used in attached to each of the overall architecture diagrams and a full definition is given below.

This application of these symbols applies to the particular diagram. For example, a system that is removed from the provisioning interface but still exists for Billing will be shown as functionally changed at the project level, removed for provisioning and unchanged for Billing.

Marking of text only applies to new or removed parameters within a message in the flow diagrams for a changed message.

Some references:

https://en.wikipedia.org/wiki/Architectural_pattern#Examples

https://en.wikipedia.org/wiki/List_of_software_architecture_styles_and_patterns

https://www.visual-paradigm.com/features/aws-architecture-diagram-tool/

Install Hashicorp Vault on a Credit card sized computer - RaspberryPi

2020-05-12T00:00:00+01:00

We want to discuss about one of growing secret service, which can be used with most of cloud services and DevOps tools. In this guide, will explain about How to Setup HashiCorp Vault on RaspberryPi.

In this blog, we're using the filesystem backend to store encrypted secrets on the local filesystem at ~/Hashicorp/vault-data.
This is suitable for local or single-server deployments that do not need to be replicated. This is not suitable for HA Setup.

cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Introduction

Vault is an open-source tool that provides a secure, reliable way to store and distribute secrets like API keys, access tokens, and passwords. Software like Vault can be critically important when deploying applications that require the use of secrets or sensitive data.

To download latest vault package, Go to Hashicorp vault downloads page and download the latest package. I am using this:

https://releases.hashicorp.com/vault/1.4.3/vault_1.4.3_linux_arm.zip

Unzip the package

unzip vault_1.4.3_linux_arm.zip

Make the vault executable to /usr/bin

sudo mv vault /usr/bin/

Checking its version.

vault -v

Finally, set a Linux capability flag on the binary. This adds extra security by letting the binary perform memory locking without unnecessarily elevating its privileges.

sudo setcap cap_ipc_lock=+ep /usr/bin/vault

Create vault data folder.

sudo mkdir ~/hashicorp/vault-data

Creating the Vault startup file

sudo useradd -r -d ~/hashicorp/vault-data -s /bin/nologin vault

Set the ownership of /vault-data to the vault user and the vault group exclusively.

sudo install -o vault -g vault -m 750 -d ~/hashicorp/vault-data

Now let’s set up Vault’s configuration file, /etc/vault.hcl

sudo vi /etc/vault.hcl
ui = true
storage "file" {
  path = "/home/theashwanik/hashicorp/vault-data""
}
listener "tcp" {
 address     = "0.0.0.0:8200"
 tls_disable = 1
}

Change ownership

sudo chown vault:vault /etc/vault.hcl
sudo chmod 640 /etc/vault.hcl

Startup script /etc/systemd/system/vault.service

sudo vi /etc/systemd/system/vault.service

[Unit]
Description=HashiCorp Vault to manage secrets
Documentation=https://vaultproject.io/docs/
After=network.target
ConditionFileNotEmpty=/etc/vault.hcl

[Service]
User=vault
Group=vault
ExecStart=/usr/bin/vault server -config=/etc/vault.hcl
ExecReload=/usr/local/bin/kill --signal HUP $MAINPID
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
AmbientCapabilities=CAP_IPC_LOCK
SecureBits=keep-caps
NoNewPrivileges=yes
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

Start the service using following command

systemctl deamon-reload  
systemctl start vault.service  
systemctl status vault.service

Preparing to Administer Vault

Add the Vault bin directory to your PATH environment variable.

export PATH=$PATH:/opt/vault/bin
echo "export PATH=$PATH:/opt/vault/bin" >> ~/.bashrc
Set environment variables for Vault


export VAULT_ADDRESS=http://192.168.1.111:8200
echo "export VAULT_ADDR=http://192.168.1.111:8200" >> ~/.bashrc

And Probably
 export VAULT_TOKEN=

**Start the command with a space. See Protip.

Attention: Be careful when you export sensitive data as environment variable using a command.
ProTip: Start your command with a space, and it will not get recorded in the command history.

Initialize Vault

There are two pieces of information that Vault will expose at initialization time that will not be available at any other point, so make sure you noted some secure place,

Initial root token: This is equivalent to root permissions to your Vault deployment, which allows the management of all Vault policies, mounts, and so on.

Unseal keys: These are used to unseal Vault when the daemon starts, which permits the Vault daemon to decrypt the backend secret stor

vim /etc/systemd/system/vault.service

Seal/Unseal

Every initialized Vault server starts in the sealed state. From the configuration, Vault can access the physical storage, but it can't read any of it because it doesn't know how to decrypt it. The process of teaching Vault how to decrypt the data is known as unsealing the Vault.

Unsealing has to happen every time Vault starts. It can be done via the API and via the command line. To unseal the Vault, you must have the threshold number of unseal keys. In the output above, notice that the "key threshold" is 3. This means that to unseal the Vault, you need 3 of the 5 keys that were generated.

Note: Vault does not store any of the unseal key shards. Vault uses an algorithm known as Shamir's Secret Sharing to split the master key into shards. Only with the threshold number of keys can it be reconstructed and your data finally accessed.

Initialize vault to get the keys.

vault operator init
Unseal Key 1: UBXbFKpvvytWeR3rUWi1k3xxxxxxxxxK8LIKtdMGvsjA
Unseal Key 2: 13sjixnJMSvNyANqwdxxxxxxxxE3OPd/izsg8nezTv3F
Unseal Key 3: /Jo+IW40WN7UQZXL6TxxxxxxxQAABhdlwth8IenTuduV
Unseal Key 4: 8YkysMXH/rsS3GOdCfW1qEwBiBk4JaKSXPjv/B0StaSF
Unseal Key 5: RRqXVkJ7o0nSsYxxxxxxxFUvvONI2meiF+E+dhssnSdO

Initial Root Token: s.VCVsxxxxxxxYMaxeYbMBUNPF0

By default, vault will be sealed. It should be unsealed with minimum of three unseal keys as shown below.

vault operator unseal UBXbFKpvvytWeR3rUWi1k3xxxxxxxxxK8LIKtdMGvsjA
vault operator unseal 13sjixnJMSvNyANqwdxxxxxxxxE3OPd/izsg8nezTv3F
vault operator unseal RRqXVkJ7o0nSsYxxxxxxxFUvvONI2meiF+E+dhssnSdO

theashwanik@ashberrypi:~/hashicorp/vault $   vault operator unseal UBXbFKpvvytWeR3rUWi1k3xxxxxxxxxK8LIKtdMGvsjA
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       634a8b1a-6d15-d4a3-740f-f6b8f01a4a37
Version            1.4.3+ent
HA Enabled         false
theashwanik@ashberrypi:~/hashicorp/vault $  vault operator unseal 13sjixnJMSvNyANqwdxxxxxxxxE3OPd/izsg8nezTv3F
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       634a8b1a-6d15-d4a3-740f-f6b8f01a4a37
Version            1.4.3+ent
HA Enabled         false
theashwanik@ashberrypi:~/hashicorp/vault $  vault operator unseal RRqXVkJ7o0nSsYxxxxxxxFUvvONI2meiF+E+dhssnSdO
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.4.3+ent
Cluster Name    vault-cluster-7944b651
Cluster ID      2573fdfa-01a5-19e1-8a20-0cd5fcc89df8
HA Enabled      false

Check the status

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.4.3+ent
Cluster Name    vault-cluster-7944b651
Cluster ID      2573fdfa-01a5-19e1-8a20-0cd5fcc89df8
HA Enabled      false

Note: Every time you restart vault or if it gets restarted during server restarts, you need to perform the unseal operation using the same unseal key.

You can also access the vault UI on port 8200 of your vault server.

http://192.168.1.111:8200/ui/

Usage

Create secrets at the kv/my-secret path.

$ vault kv put kv/my-secret value="s3c(eT"
Success! Data written to: kv/my-secret

Read the secrets at kv/my-secret.

$ vault kv get kv/my-secret

==== Data ====
Key      Value
---      -----
value    s3c(eT

Delete the secrets at kv/my-secret.

$ vault kv delete kv/my-secret
Success! Data deleted (if it existed) at: kv/my-secret

List existing keys at the kv path.

$ vault kv list kv/

Keys
----
hello

Disable a Secrets Engine

When a secrets engine is no longer needed, it can be disabled. When a secrets engine is disabled, all secrets are revoked and the corresponding Vault data and configuration is removed.

$ vault secrets disable kv/
Success! Disabled the secrets engine (if it existed) at: kv/

Note that this command takes a PATH to the secrets engine as an argument, not the TYPE of the secrets engine.

Dynamic Secret Engines:

vault secrets enable -path=aws aws
Success! Enabled the aws secrets engine at: aws/

Getting help

vault path-help aws

DESCRIPTION

The AWS backend dynamically generates AWS access keys for a set of
IAM policies. The AWS access keys have a configurable lease set and
are automatically revoked at the end of the lease.

After mounting this backend, credentials to generate IAM keys must
be configured with the "root" path and policies must be written using
the "roles/" endpoints before any access keys can be generated.

PATHS

The following paths are supported by this backend. To view help for
any of the paths below, use the help command with any route matching
the path pattern. Note that depending on the policy of your auth token,
you may or may not be able to access certain paths.

    ^(creds|sts)/(?P\w(([\w-.@]+)?\w)?)$
        Generate AWS credentials from a specific Vault role.

    ^config/lease$
        Configure the default lease information for generated credentials.

    ^config/root$
        Configure the root credentials that are used to manage IAM.

    ^config/rotate-root$
        Request to rotate the AWS credentials used by Vault

    ^roles/(?P\w(([\w-.@]+)?\w)?)$
        Read, write and reference IAM policies that access keys can be made for.

    ^roles/?$
        List the existing roles in this backend

Authentication

vault token create
Key                  Value
---                  -----
token                s.7vM3kUTFSNxxxxxxxxxf4f8R9
token_accessor       Dtuk4LtxxxxxxxrEDNXiB5EZ
token_duration       -
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

The token is created and displayed here as s.7vM3kUTFSNxxxxxxxxxf4f8R9. Each token that Vault creates is unique.

vault login s.7vM3kUTFSNxxxxxxxxxf4f8R9
WARNING! The VAULT_TOKEN environment variable is set! This takes precedence
over the value set by this command. To use the value set by this command,
unset the VAULT_TOKEN environment variable or set it to the token displayed
below.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.7vM3kUTFSNxxxxxxxxxf4f8R9
token_accessor       Dtuk4LtxxxxxxxrEDNXiB5EZ
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

When a token is no longer needed it can be revoked.

Revoke the first token you created.

$ vault token revoke s.7vM3kUTFSNxxxxxxxxxf4f8R9
Success! Revoked token (if it existed)

The token has been revoked.

That's it folks.

Few screenshots

Home page of Hashicorp UI once you login successfully.

Secrets screen - where all your secret engines and secrets can be seen.

Secrets screen - configuration of KV secret engine

References

More information here by hashicorp

Issue

Google 2FA with OpenVPN on AWS

2020-04-02T00:00:00+01:00

OpenVPN Google authenticator setup

How to enable Google Authenticator**

The Access Server supports the Google Authenticator multi-factor authentication system, but it is not enabled by default. It can be enabled globally via the admin web service (AS 2.7.4 and older) or via the .Authentication" section (AS 2.7.5 and newer) or via the command line with the command line examples given below. It is also possible to enable or disable the requirement for a Google Authenticator per user or per group on the command line. This can be important if for example for some reason a client device making a VPN connection is unable to provide the Google Authenticator key by itself.

Command line configuration parameters

Disable Google Authenticator globally for all users and groups (the default):

./sacli --key "vpn.server.google_auth.enable" --value "false" ConfigPut
./sacli start

Enable Google Authenticator globally for all users and groups:

./sacli --key "vpn.server.google_auth.enable" --value "true" ConfigPut
./sacli start

Disable Google Authenticator for a specific user or group:

./sacli --user  --key "prop_google_auth" --value "false" UserPropPut

Enable Google Authenticator for a specific user or group:

./sacli --user  --key "prop_google_auth" --value "true" UserPropPut

Undo an enable/disable override for Google Authenticator on a group or user, so that it inherits the setting instead:

./sacli --user  --key "prop_google_auth" UserPropDel

To unlock an already scanned and locked secret for a user, so the user can obtain/scan it again:

./sacli --user  --lock 0 GoogleAuthLock

To manually lock a secret key, for example when you as administrator have already set up the user.s device yourself:

./sacli --user  --lock 1 GoogleAuthLock

To generate a new secret key and unlock it so the user can enroll anew:

./sacli --user  --lock 0 GoogleAuthRegen

To generate a new secret key and lock it so the user must obtain the secret key from the server administrator:

./sacli --user  --lock 1 GoogleAuthRegen

The GoogleAuthLock and GoogleAuthRegen functions that actually handle these two keys, which can also be edited manually:

./sacli --user  --key "pvt_google_auth_secret" --value  UserPropPut

./sacli --user  --key "pvt_google_auth_secret_locked" --value  UserPropPut

Where must be a 16 character alphanumerical value in capitals and must be known at the Google Authenticator device/application to generate the 6 digit codes, and the value must be either 1 or 0, indicating that the code is scanned and must now be used by the user, or is awaiting enrollment by the user.

cd /usr/local/openvpn_as/scripts

sudo ./confdba -us -p joe _#display info about a user_

{

  "joe": {

  "access_to.0": "+NAT:10.0.0.0/8",

  "pvt_google_auth_secret": "Z********B", _#this is GoogleAuth MFA secret_token that a user scans as QR code_

  "pvt_google_auth_secret_locked": "false",

  "pvt_password_digest": "30******bb71",

  "type": "user_compile"

 }

}

sudo ./confdba -u -m -k pvt_google_auth_secret_locked -v false -p joe  _#unlock locked out user_

_#Disable/enable Google Authenticator for a specific user or group:   _

./sacli --user  --key "prop_google_auth" --value "false" UserPropPut _#disable_

./sacli --user  --key "prop_google_auth" --value "true" UserPropPut _#enable_

_#Undo an enable/disable override for Google Authenticator on a group or user, so that it inherits the setting instead_

./sacli --user  --key "prop_google_auth" UserPropDel

_#To unlock an already scanned and locked secret for a user, so the user can obtain/scan it again_

./sacli --user  --lock 0 GoogleAuthLock

_#To manually lock a secret key, for example when you as administrator have already set up the user.s device yourself_

./sacli --user  --lock 1 GoogleAuthLock

_#To generate a new secret key and lock or leave it unlocked_

./sacli --user  --lock 0 GoogleAuthRegen _#unlocked, user can scan_

./sacli -u  joe  GoogleAuthRegen _#regenerate Google token, so a user can scan QR code again_

The GoogleAuthLock and GoogleAuthRegen functions that actually handle these two keys, which can also be edited manually

./sacli --user  --key "pvt_google_auth_secret" --value  UserPropPut

./sacli --user  --key "pvt_google_auth_secret_locked" --value   UserPropPut

Logs

tail -f /var/log/openvpnas.log

When new MFA/Google secret has been generated user need to login to Access Server, scann QR code, then download the Connection Client that the bundle contains the new user settings; this will enable VPN login.

AWS SSM Document:

Handy AWS Sytems Manager Document that can be used to unlock Google Authenticator for a user. Simply add this Document to Systems Manager and Run it with an instance and the username of the user to unlock. This requires installation of the SSM agent on each OpenVPN instance. You'll probably need to read up on the AWS Systems Manager docs but it is well worth it for this and a whole host of other use cases.

{

      "schemaVersion": "2.2",
      "description": "Unlock the Google Authenticator for a given Username. After doing this, the user must login to the OpenVPN server with their browser and scan the barcode.",
      "parameters": { 
          "Username": {
          "description": "Username of the user to unlock",
          "minChars": 3,
          "type": "String"
          }
      },
      "mainSteps": [
          {
              "action": "aws:runShellScript",
              "name": "OpenVPNASUnlockGoogleAuthenticator",
              "inputs": {
                  "runCommand": [
                      "#!/bin/bash",
                      "cd /usr/local/openvpn_as/scripts",
                      "sudo ./sacli -u {REMOVE_THIS_TEXT% Username %} --lock 0 GoogleAuthLock"
                      ]
                  },
              "precondition":{
              "StringEquals":[
                      "platformType",
                      "Linux"
                      ]

              }

          }
      ]

  }

Issues:

Sometimes you would wonder , why is my EC2 instance not appearing under Managed Instances in the Systems Manager console?

A managed instance is an EC2 instance that is configured for use with Systems Manager. Managed instances can use Systems Manager services such as Run Command, Patch Manager, and Automation workflows.

Instances must meet the following prerequisites to be managed instances:

Have the AWS Systems Manager Agent (SSM Agent) installed and running.
Have connectivity with Systems Manager endpoints using the SSM Agent.
Have the correct AWS Identity and Access Management (IAM) role attached.

Have connectivity to the instance metadata service

Resolution

SSM Agent is installed and running on the instance

Latest Ubuntu 18.04 systems that use snap:

$ sudo snap services amazon-ssm-agent

Service  Startup  Current  Notes

amazon-ssm-agent.amazon-ssm-agent  enabled  active  -

Verify connectivity to Systems Manager endpoints on port 443 To test connectivity to endpoints from port 443, use the telnet command. The following example shows how to test connectivity to endpoints in the us-east-1 Region.

telnet ssm.us-east-1.amazonaws.com 443

telnet ec2messages.us-east-1.amazonaws.com 443

telnet ssmmessages.us-east-1.amazonaws.com 443

Verify that the correct IAM role is attached to the instance To use APIs to call a Systems Manager endpoint, the correct IAM role must be attached to the instance. Make sure that the IAM role has the AWS managed policy AmazonSSMManagedInstanceCore attached to it. If you are using a custom IAM policy, make sure that the permissions found under AmazonSSMManagedInstanceCore are used in your custom policy. Also, make sure that the trust policy of the IAM role allows ec2.amazonaws.com to assume this role.
Verify connectivity to the instance metadata service SSM Agent must be able to communicate with the instance metadata service in order to get necessary information about the instance. To test this connection, use the telnet command.

telnet 169.254.169.254 80

References

https://openvpn.net/vpn-server-resources/google-authenticator-multi-factor-authentication/

https://openvpn.net/vpn-server-resources/additional-security-command-line-options/

https://aws.amazon.com/premiumsupport/knowledge-center/systems-manager-ec2-instance-not-appear/

https://docs.aws.amazon.com/systems-manager/latest/userguide/agent-install-ubuntu.html

Open VPN with LetsEncrypt certificate

2020-02-09T00:00:00+00:00

LetsEncrypt HTTPS Certificates for OpenVPN AS (Access Server)

So you want secure transport using ssl certificate for your openvpn Access server ( the admin GUI) To load a new HTTPS certificate for OpenVPN AS (Access Server), you.ll want to use the ./usr/local/openvpn_as/scripts/confdba command. This can be combined with a Lets Encrypt client to obtain free a HTTPs certificate for the AS web server.

Step 1 . Installing Certbot

The first step to using LetsEncrypt to obtain an SSL certificate is to install the certbot software on your server.

First, add the repository:

sudo add-apt-repository ppa:certbot/certbot ````

You'll need to press ENTER to accept. Afterwards, update the package list to pick up the new repository.s package information:

sudo apt-get update

And finally, install Certbot with apt-get:

sudo apt-get install python-certbot-nginx

The certbot LetsEncrypt client is now ready to use.

Step 2 . Setting up Nginx - Optional Not needed for OpenVPN

Certbot can automatically configure SSL for Nginx, but it needs to be able to find the correct server block in your config. It does this by looking for a server_name directive that matches the domain you.re requesting a certificate for. If you.re starting out with a fresh Nginx install, you can update the default config file:

sudo nano /etc/nginx/sites-available/default Find the existing server_name line:

/etc/nginx/sites-available/default server_name localhost; Replace localhost with your domain name:

/etc/nginx/sites-available/default server_name example.com www.example.com; Save the file and quit your editor. Verify the syntax of your configuration edits with:

sudo nginx -t If that runs with no errors, reload Nginx to load the new configuration:

sudo service nginx reload Certbot will now be able to find the correct server block and update it. Now we.ll update our firewall to allow HTTPS traffic.

Step 3 . Obtaining an SSL Certificate

Certbot provides a variety of ways to obtain SSL certificates, through various plugins. The Nginx plugin will take care of reconfiguring Nginx and reloading the config whenever necessary:

sudo certbot --nginx -d example.com -d www.example.com This runs certbot with the --nginx plugin, using -d to specify the names we.d like the certificate to be valid for.

If this is your first time running certbot, you will be prompted to enter an email address and agree to the terms of service. After doing so, certbot will communicate with the Let.s Encrypt server, then run a challenge to verify that you control the domain you.re requesting a certificate for.

If that.s successful, certbot will ask how you.d like to configure your HTTPS settings:

Output

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.

1: No redirect - Make no further changes to the webserver configuration. 2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for new sites, or if you're confident your site works on HTTPS. You can undo this

change by editing your web server's configuration.

Select the appropriate number [1-2] then [enter] (press 'c' to cancel): Select your choice then hit ENTER. The configuration will be updated, and Nginx will reload to pick up the new settings. certbot will wrap up with a message telling you the process was successful and where your certificates are stored:

Output IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at /etc/letsencrypt/live/example.com/fullchain.pem. Your cert will expire on 2017-10-23. To obtain a new or tweaked version of this certificate in the future, simply run certbot again with the "certonly" option. To non-interactively renew all of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal. - If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le Your certificates are now downloaded, installed, and configured. Try reloading your website using https:// and notice your browser.s security indicator. It should represent that the site is properly secured, usually with a green lock icon. If you test your server using the SSL Labs Server Test, it will get an A grade.

Step 4 . Verifying Certbot Auto-Renewal

Let.s Encrypt.s certificates are only valid for ninety days. This is to encourage users to automate their certificate renewal process. The certbot package we installed takes care of this for us by running .certbot renew. twice a day via a systemd timer. On non-systemd distributions this functionality is provided by a script placed in /etc/cron.d. This task runs twice a day and will renew any certificate that.s within thirty days of expiration.

To test the renewal process, you can do a dry run with certbot:

sudo certbot renew --dry-run

Commands Used

Proxies Explained or Demystified

2019-11-02T00:00:00+00:00

Proxy is Essentially, a middle man. When dealing with computers the concept is largely the same. A web proxy is simply a bit of software that will relay a HTTP request for you.

This post is about - Proxies Made Easy or as my good friend at very good security say - "Proxies Demystified.."

What is a Proxy

HTTP Proxies are an essential component when using the internet day to day - load balancers, routers, content accelerators, content protection systems, these are all simple examples of web proxies and they all act as intermediaries to send your HTTP requests where they need to go, anonymize requests, handle routing of traffic, speed up the net, and many other uses.

Google defines a proxy as

the authority to represent someone else

When it comes down to it, most web proxies fall into two camps:

Reverse Proxies - A reverse proxy is usually an internal-facing proxy used as a front-end to control and protect access to servers on a private network. A reverse proxy commonly also performs tasks such as load-balancing, authentication, decryption or caching.
Forward Proxies - A forward proxy is an Internet-facing proxy used to retrieve from a wide range of sources. Let.s look at these two types in more detail

What is a Reverse Proxy

You.re probably using a reverse proxy in order to view this content. When you make a request to the server that serves this blog post it will pass through a load balancer. This load balancer is a type of reverse proxy. Reverse proxies will sit in front of one of more servers and distribute requests to these servers. Common examples of these would be Nginx.s proxy_pass module, HAProxy, Squid, and AWS. ELB.

Let.s look at an example of a reverse proxy:

Reverse Proxy

You can see in this graphic that the reverse proxy receives a request from a client on the internet and retrieves the requested resources from one of more servers that sit behind it. To the client there is no knowledge required of the servers (often called upstream servers) that serve the original content and they can be changed as required without any outside knowledge. The reverse proxy handles that information.

As part of this handling of the request the reverse proxy will often provide additional value such as terminating SSL, performing authentication and/or authorization, accelerating (caching or compressing) content or rewriting the request and/or response.

The word .reverse. in the name reverse proxy has no special meaning, it.s just used as the inverse of forward proxy which actually has a meaning as you.ll read shortly.

What is a Forward Proxy

Forward proxies are commonly used to control traffic leaving networks. When you send a request via the proxy it will .forward. your request on to the requested website, hence the name .Forward Proxy..

A common job of a forward proxy is to both control access to the internet by inspecting certain attributes as the request passes through it. If you.re on a corporate network and they prohibit you from going to a social network such as facebook.com, this will often be the job of a forward proxy. The forward proxy is able to inspect the host of the request and, since on a corporate network traffic is often mandated to flow through the proxy, it will deny any requests that use the prohibited host.

A similar implementation to the above will scan outbound content of the payload as it passes through the proxy. This can be used for a variety of data protection applications e.g., for data loss prevention; or scan content for malicious software.

Another common use-case for a forward proxy is to anonymize where the request originally came from.

Let.s look at an example of a forward proxy:

Forward Proxy

As you can see it sits in between requests from the user to the internet. As such, when the forward proxy sends a request to a host the host computer will see the IP address of the forward proxy, not of the user. This is commonly used to perform IP anonymization and is a major feature of VPNs.

Layer 7 versus layer 3

Most of the time .proxy. refers to a layer-7 application on the OSI reference model. However, another way of proxying is through layer-3 and is known as Network Address Translation (NAT). The difference between these two proxy technologies is the layer in which they operate, and the procedure to configuring the proxy clients and proxy servers.

Layer-7 proxies are more suitable if you.re inspecting the content of the payload to perform routing or otherwise manipulating the payload.

References
Youtube link
Quora - Whats-the-difference-between-a-reverse-proxy-and-forward-proxy
Jscape - Forward-Proxy-vs-Reverse-Proxy
Citrix - reverse-vs-forward-proxy
Google - forward+versus+reverse+proxy
Apache - mod_proxy.html#forwardreverse

Disclaimer: **This is Collaboration post with verygoodsecurity.com and this post is directly published on collaboration request from Eugene - genewalz.xxxx@gmail.com
Original article

Few Tweaks in Diaspora installation

2019-07-15T19:41:00+01:00

Recently I wanted to checkout the Diaspora project, and to my despair I faced lot of problems installing it in Windows OS.
I could not install it on Windows, and I did'nt want to invest lot of time , as it was not important and I was trying this only for killing my time. However, I installed it on a VM (Ubuntu) and faced a few problems.

This post is about keeping a note of what I faced and use it in future (if needed).

1. Issue with installing rmagick:

rmagick installation issue

ERROR:  Error installing rmagick:
    ERROR: Failed to build gem native extension.

/usr/local/bin/ruby extconf.rb
checking for Ruby version >= 1.8.5... yes
checking for gcc... yes
checking for Magick-config... no
Can't install RMagick 2.13.1. Can't find Magick-config in /usr/local/bin:/bin:
/sbin:/usr/bin:/usr/sbin:/usr/libexec
.
.
.
Gem files will remain installed in /usr/local/lib/ruby/gems/1.8/gems/rmagick-2.13.1 for inspection.
Results logged to /usr/local/lib/ruby/gems/1.8/gems/rmagick-2.13.1/ext/RMagick/gem_make.out
Building native extensions.  This could take a while....

I found a solution here but it did not work for me.

To solve the rmagick issue, run the following commands:

Run the following commands to fix issues with rmagick

sudo apt-get install libmagickcore-dev libmagickwand-dev  
sudo gem install rmagick

2. Issue with installing typhoeus:

rmagick installation issue

If you get this:

Installing typhoeus (0.3.3) with native extensions
/usr/local/lib/site_ruby/1.8/rubygems/installer.rb:483:in `build_extensions': 
ERROR: Failed to build gem native extension. (Gem::Installer::ExtensionBuildError)

/usr/bin/ruby1.8 extconf.rb 
checking for curl/curl.h in /opt/local/include,/opt/local/include/curl,/usr/include/curl,
/usr/include,/usr/include/curl,/usr/local/include/curl... no
need libcurl

You can fix it by installing the libcurl3-dev package:

commands to fix issues with typhoeus

sudo apt-get install libcurl3-dev
sudo apt-get install typhoeus

I did not want to use mysql2 adapter for diaspora so I had to install postgreSQL (Since sqlite had problems with some length of index names)

installing postgresql in linux

sudo apt-get install postgresql-9.1

If you find issue in some gem which depends on curl, then you must install curl first.

installing curl

sudo apt-get install curl

That's all folks.
au revoir

Simple Port Forwarding script

2019-05-15T19:29:00+01:00

Port forwarding allows remote computers (for example, computers on the Internet) to connect to a specific computer or service within a private local-area network (LAN).
Source:Wiki

Here is my version of local port forwarding. I created a script for finding me a local unused port and then forward all traffic on that local port to some remote server on a predefined port.

// Create a script called portForward.sh
// (This has to be run from the Server where you need to setup port forwarding)
//
Port=32000  // Set initial port to start lookup from.
while netstat -atwn | grep "^.*:${Port}.*:\*\s*LISTEN\s*$"
do
  echo "Port $Port is already used"
  Port=$(( ${Port} + 1 ))
done
echo "Looks like you get port ${Port}."  // Say you got 32003

//ssh -f userOfRemoteServer@remoteserverIP -L LocalhostIPaddress:localport:RemotehostIPaddress:remoteport -N
ssh -f user@10.145.2.233 -L 10.115.13.108:${Port}:10.145.2.233:8000 -N

The above script will forward all the traffic on 10.115.13.108:32003 to 10.145.2.233:8000.

Please note that you will need a user to be existing on 10.145.2.233. The script will ask you for the password.

Hope this helps someone.

HAProxy for load balancing

2019-03-25T00:00:00+00:00

HAProxy, which stands for High Availability Proxy, is a popular open source software TCP/HTTP Load Balancer and proxying solution which can be run on Linux, Solaris. Improves the performance and reliability of a server environment by distributing the traffic across servers (e.g. web, application, database). It is used in many high-profile environments websites.

In this guide, I will provide a general overview of what HAProxy is, basic load-balancing terminology, and examples of how it might be used to improve the performance and reliability of your own server environment.

This post is about some setup required for HAProxy

HAProxy can balance requests between any application that can handle HTTP or even TCP requests.

Install HAProxy on Pi

Credit goes to load-balancing-with-haproxy

sudo apt-get update
sudo apt-get install -y haproxy

HAProxy Configuration


HAProxy configuration can be found at /etc/haproxy/haproxy.cfg. Here's what we'll likely see by default:

sudo vi /etc/haproxy/haproxy.cfg


global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        # An alternative list with additional directives can be obtained from
        #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http


frontend www-http
        bind *:80
        mode http
        default_backend http-nodes

frontend www-https
        bind *:443 ssl crt /etc/ssl/private/letsencrypt-ForHaproxy.pem
        reqadd X-Forwarded-Proto:\ https
        acl letsencrypt-acl path_beg /.well-known/acme-challenge/
        use_backend letsencrypt-backend if letsencrypt-acl
        default_backend http-nodes

backend letsencrypt-backend
        server letsencrypt 127.0.0.1:54321

backend http-nodes
        mode http
        balance roundrobin
        option forwardfor
        http-request set-header X-Forwarded-Port %[dst_port]
        http-request add-header X-Forwarded-Proto https if { ssl_fc }
        option httpchk HEAD / HTTP/1.1\r\nHost:localhost

        #redirect scheme https if !{ ssl_fc }
        server web01 127.0.0.1:9000 check
        server web02 127.0.0.1:9001 check


listen stats
        bind *:1936
        stats enable
        stats uri /
        stats hide-version
        stats auth username:password

LetsEncrypt Certificate

Use acme-nginx to generate letsencrypt site for your site.

sudo cat /etc/ssl/private/letsencrypt-domain.key /etc/ssl/private/letsencrypt-domain.pem > /etc/ssl/private/letsencrypt-ForHaproxy.pem
sudo mv letsencrypt-ForHaproxy.pem /etc/ssl/private/
sudo chown -R user:group /etc/ssl/private/letsencrypt-ForHaproxy.pem

Resources

Who is using HAProxy
HAProxy Introduction
https://serversforhackers.com/c/load-balancing-with-haproxy
https://serversforhackers.com/c/using-ssl-certificates-with-haproxy

PowerShell script to parse multiple Robocopy log files

2019-01-12T10:56:00+00:00

Recently came across a problem of parsing a robocopy logs at work. I found this helpful script, so reblogging for my future reference.

This is a PowerShell script that can be used to parse multiple Robocopy log files.

Reference:See more here

All credits to the respective authors. I am keeping this for future reference. Hope this is fine.

Robocopy Parser script

function Analyse-RC_Log {
    <#
    .SYNOPSIS
        Robocopy log analyser
    .DESCRIPTION
        analysing the robocopy logs that are generated with the /LOG option.
        It has two modes, anaylsing the summary of log files and analyse the full log.
        The report is saved as a CSV file.
        Returns an custom object containing a RC summary like property, a CSV property.
        During script process the Culture of the script is changed to en-US for date and number interpretation / calculations

        Script is based on Robocopy log analyser from http://www.chapmancentral.co.uk/cloudy/2013/02/23/parsing-robocopy-logs-in-powershell/

    .EXAMPLE
        >Analyse-RC_Log -SourcePath d:\RC_logdir -ExcelCSV -fp -unitsize GB
        Analyse log files in d:\RC_logdir, use a semicolon in the CSV file (conform MS Excel). Parse the complete log files and report al Bytes sizes as GB

    .EXAMPLE
        >$Result=Analyse-RC_Log -SourcePath d:\RC_logdir -ExcelCSV -fp -unitsize GB
    >& $Result.ReportFileName
    >$Result.Summary | out-gridview

    opens the CSV file and show the summary in a powershell gridview

    .PARAMETER fp
        File Parsing. Parse the complete file instead of the heather and footer

    .PARAMETER SourcePath
        Path where the Robocopy logs are saved.

    .PARAMETER ExcelCSV
        Use a semicolon as seperator

    .Parameter UnitSize
        Report al Byte sizes  in given unit size.

    .Link
        http://www.chapmancentral.co.uk/cloudy/2013/02/23/parsing-robocopy-logs-in-powershell/

    .NOTES
        File Name      : Analyse-RC_Log.ps1
        Author         : B. Lievers
        Prerequisite   : PowerShell V2 over Vista and upper.
        Copyright 2015 - Bart Lievers

    #>

    [CmdletBinding()]

    param(
        [parameter(Position=0,Mandatory=$true,ValueFromPipeline=$false,HelpMessage='Source Path with no trailing slash')][string]$SourcePath,
        [switch]$fp,
        [Switch]$ExcelCSV,
        [Parameter(HelpMessage='Unit size, default is Bytes when parameter is not present')][ValidateSet("B","GB","KB","MB","TB")][string]$UnitSize
        )

    begin {
        [System.Globalization.CultureInfo]$culture=[System.Globalization.CultureInfo]("en-US")
        $OldCulture = [System.Threading.Thread]::CurrentThread.CurrentCulture
        trap 
        {
            [System.Threading.Thread]::CurrentThread.CurrentCulture = $OldCulture
        }
        [System.Threading.Thread]::CurrentThread.CurrentCulture = $culture
        Write-Verbose ("Changing Locale from "+$oldCulture.Name+" to "+$culture.Name)

        write-host "Robocopy log parser. $(if($fp){"Parsing file entries"} else {"Parsing summaries only, use -fp to parse file entries"})"

        $ElapsedTime = [System.Diagnostics.Stopwatch]::StartNew()
        $refreshrate=1 # progress counter refreshes this often when parsing files (in seconds)

        #region initialize header fields
        # These summary fields always appear in this order in a robocopy log
        $HeaderParams = @{
            "04|Started" = "date";  
            "01|Source" = "string";
            "02|Dest" = "string";
            "03|Options" = "string";
            "07|Dirs" = "counts";
            "08|Files" = "counts";
            "09|Bytes" = "counts";
            "10|Times" = "counts";
            "05|Ended" = "date"
            #"06|Duration" = "string"
        }
        #-- summary fields for file tag statistics during file parsing, swich -fp
        $fileTags=@{ 
            "01|MISMATCH" = ""
            "02|EXTRA file" = ""
            "03|New File" = ""
            "04|lonely" = ""
            "05|Newer" = ""
            "06|Newer XN" = ""
            "07|Older" = ""
            "08|Older XO" = ""
            "09|Changed" = ""
            "10|Changed XC" = ""
            "11|Tweaked" = ""
            "12|Same IS" = ""
            "13|Same" = ""
            "14|attrib" = ""
            "15|named" = ""
            "16|large" = ""
            "17|small" = ""
            "18|too old" = ""
            "19|too new" = ""
            "20|New Dir"= ""
        }
        #-- summary fields for directory tag statistics during file parsing, swich -fp
        $DirTags=@{ 
            "01|MISMATCH" = ""
            "02|Extra Dir" = ""
            "03|New Dir" = ""
            "04|lonely" = ""
            "05|named" = ""
            "06|junction" = ""
            "07|exist" = ""
        } 

        $ProcessCounts = @{
            "Processed" = 0;
            "Error" = 0;
            "Incomplete" = 0
        }
        #endregion

        #-- Default the CSV delim is a comma, when using CSV for Excel a semicolon is needed as delimmiter
        if ($ExcelCSV) { $Delim=";"} else {$Delim=","}
         #-- ASCII tab character
        $tab=[char]9

         #-- Get list of files to analyse
        $files=get-childitem $SourcePath
        #-- let's start writing, shall we ? 
        $writer=new-object System.IO.StreamWriter("$(get-location)\robocopy-$(get-date -format "dd-MM-yyyy_HH-mm-ss").csv")


        #region private functions

        function Get-Tail{
            <#
            .SYNOPSIS
                Get tail of file
            .EXAMPLE
                >Get-Tail $reader 20
            .PARAMETER reader
                an IO stream file object
            .PARAMETER count
                Number of rows to collect
            #>
            Param (
                [object]$reader, 
                [int]$count = 10
                )

            $lineCount = 0
            [long]$pos = $reader.BaseStream.Length - 1

            while($pos -gt 0) {
                $reader.BaseStream.position=$pos

                # 0x0D (#13) = CR
                # 0x0A (#10) = LF
                if ($reader.BaseStream.ReadByte() -eq 10) {
                    $lineCount++
                    if ($lineCount -ge $count) { break }
                    }
                $pos--
                } 

            # tests for file shorter than requested tail
            if ($lineCount -lt $count -or $pos -ge $reader.BaseStream.Length - 1) {
                $reader.BaseStream.Position=0
            } else {
                # $reader.BaseStream.Position = $pos+1
            }

            $lines=@()
            while(!$reader.EndOfStream) {
                $lines += $reader.ReadLine()
            }
            return $lines
        }

        function Get-Top {
            <#
            .SYNOPSIS
                Get top of file
            .EXAMPLE
                >Get-Top $reader 20
            .PARAMETER reader
                an IO stream file object
            .PARAMETER count
                Number of rows to collect
            #>
            Param(
                [object]$reader,
                [int]$count = 10
            )

            $lines=@()
            $lineCount = 0
            $reader.BaseStream.Position=0
            while(($linecount -lt $count) -and !$reader.EndOfStream) {
                $lineCount++
                $lines += $reader.ReadLine()        
            }
            return $lines
        }

        function Remove-Key{
            <#
            .SYNOPSIS
                Return the name without the ID
            .EXAMPLE
                >Remove-Key -name "01|example"
            .PARAMETER name
                a string where the ID needs to be stripped
            #>
            Param ( $name 
            )
            if ( $name -match "|") {
                return $name.split("|")[1]
            } else {
                return ( $name )
            }
        }

        function Get-Value{
            <#
            .SYNOPSIS
                Get the value of a RC line
            .EXAMPLE
                >Get-Value -line " filecount : 35555" -variable "Filecount"
                Returns 35555
            .PARAMETER line
                A RC log string
            .PARAMETER variable
                The variable that needs to be extracted
            #>
            Param(
                $line,
                $variable
            )
            if ($line -like "*$variable*" -and $line -like "* : *" ) {
                $result = $line.substring( $line.IndexOf(":")+1 )
                return $result 
            } else {
                return $null
            }
        }

        function UnBodge-Date{
            <#
            .SYNOPSIS
                Convert Robocopy date to a usable format
            .EXAMPLE
                >UnBodge-Date -dt "Sat Feb 16 00:16:49 2013"
                Returns 16-02-2013 00:16:49 in Locale format
            .PARAMETER dt
            #>
            Param(
                $dt
            )
            # Fixes RoboCopy botched date-times in format Sat Feb 16 00:16:49 2013
            if ( $dt -match ".{3} .{3} \d{2} \d{2}:\d{2}:\d{2} \d{4}" ) {
                $dt=$dt.split(" ")
                $dt=$dt[2]+"/"+$dt[1]+"/"+$dt[4],$dt[3]
                $dt -join " " | Out-Null
            }
            if ( $dt -as [DateTime] ) {
                return(get-date $dt -format "dd/MM/yyy HH:mm:ss")
            } else {
                return $null
            }
        }

        function Unpack-Params{
            <#
            .SYNOPSIS
                Unpacks file count bloc in the format
                 Dirs :      1827         0      1827         0         0         0
                Files :      9791         0      9791         0         0         0
                Bytes :  165.24 m         0  165.24 m         0         0         0
                Times :   1:11:23   0:00:00                       0:00:00   1:11:23
                Parameter name already removed
            .EXAMPLE
                >UnBodge-Date -dt "Sat Feb 16 00:16:49 2013"
                Returns 16-02-2013 00:16:49 in Locale format
            .PARAMETER params

            #>
            Param(
                $params
            )

            if ( $params.length -ge 58 ) {
                $params = $params.ToCharArray()
                $result=(0..5)
                for ( $i = 0; $i -le 5; $i++ ) {
                    $result[$i]=$($params[$($i*10 + 1) .. $($i*10 + 9)] -join "").trim()
                }
                #$result=$result -join ","
                $result=$result -join $Delim
            } else {
                #$result = ",,,,,"
                $result=$Delim+$Delim+$Delim+$Delim+$Delim
            }
            return $result
        }
    #endregion        
    } #-- end of Begin

    Process{
    $sourcecount = 0
        $targetcount = 1

        #region construct the header line of the CSV
    $writer.Write("File")
    $fields="File"
    foreach ( $HeaderParam in $HeaderParams.GetEnumerator() | Sort-Object Name ) {
        if ( $HeaderParam.value -eq "counts" ) {
            $tmp="~ Total"+$Delim+"~ Copied"+$Delim+"~ Skipped"+$Delim+"~ Mismatch"+$Delim+"~ Failed"+$Delim+"~ Extras"
            if ((Remove-Key $headerparam.name) -match "Bytes") {
                #-- if column header is a Bytes Column then match it to the unitsize
                if ($UnitSize -and $UnitSize -ne "B") {
                    #-- change the Bytes header according to the unitsize, Unitsize is GB ==> header is GBytes
                    $tmp=$tmp.replace("~",$UnitSize.Substring(0,1)+ "$(Remove-Key $headerparam.name)")
                } else {
                    $tmp=$tmp.replace("~","$(Remove-Key $headerparam.name)")
                }
            } else {
                $tmp=$tmp.replace("~","$(Remove-Key $headerparam.name)")
            }
            $fields=$fields+$Delim+"$($tmp)"
            $writer.write($Delim+"$($tmp)")
        } else {
            $writer.write($Delim+"$(Remove-Key $HeaderParam.name)")
            $fields=$fields+$Delim+"$(Remove-Key $HeaderParam.name)"
        }
    }

    if($fp){
        $writer.write($Delim+"Scanned"+$Delim+"Newest"+$Delim+"Oldest")
        $fields=$fields+$Delim+"Scanned"+$Delim+"Newest"+$Delim+"Oldest"
        foreach ($fileTag in $filetags.GetEnumerator() | Sort-Object Name ) {
            $writer.write($Delim+$filetag.name.split("|")[1])
             $fields=$fields+$delim+$filetag.name.split("|")[1]
                }
        foreach ($DirTag in $Dirtags.GetEnumerator() | Sort-Object Name ) {
            $writer.write($Delim+"DIR "+$DirTag.name.split("|")[1])
             $fields=$fields+$delim+"DIR "+$DirTag.name.split("|")[1]
        }
    }
     # EOL
    $writer.WriteLine()
    #endregion

        $table=$fields
        $filecount=0

        # Enumerate the files
        foreach ($file in $files) {  
            $filecount++
            write-host "$filecount/$($files.count) $($file.name) ($($file.length) bytes)"
            $results=@{}
            #-- open the file
            $Stream = $file.Open([System.IO.FileMode]::Open, 
                           [System.IO.FileAccess]::Read, 
                            [System.IO.FileShare]::ReadWrite) 
            $reader = New-Object System.IO.StreamReader($Stream) 

            $HeaderFooter = Get-Top $reader 16

            if ( $HeaderFooter -match "ROBOCOPY     ::     Robust File Copy for Windows" ) {
                #-- file has valid header
                if ( $HeaderFooter -match "Files : " ) {
                    $HeaderFooter = $HeaderFooter -notmatch "Files : "
                }

                [long]$ReaderEndHeader=$reader.BaseStream.position

                $Footer = Get-Tail $reader 16
                #check footer of file
                $ErrorFooter = $Footer -match "ERROR \d \(0x000000\d\d\) Accessing Source Directory"
                if ($ErrorFooter) {
                    $ProcessCounts["Error"]++
                    write-host -foregroundcolor red "`t $ErrorFooter"
                } elseif ( $footer -match "---------------" ) {
                    $ProcessCounts["Processed"]++
                    $i=$Footer.count
                    while ( !($Footer[$i] -like "*----------------------*") -or $i -lt 1 ) { $i-- }
                    $Footer=$Footer[$i..$Footer.Count]
                    $HeaderFooter+=$Footer
                } else {
                    $ProcessCounts["Incomplete"]++
                    #write-host -foregroundcolor yellow "`t Log file $file is missing the footer and may be incomplete"
                    write-warning "`t Log file $file is missing the footer and may be incomplete"
                }

                foreach ( $HeaderParam in $headerparams.GetEnumerator() | Sort-Object Name ) {
                    $name = "$(Remove-Key $HeaderParam.Name)"
                    $tmp = Get-Value $($HeaderFooter -match "$name : ") $name
                    if ( $tmp -ne "" -and $tmp -ne $null ) {
                        switch ( $HeaderParam.value ) {
                            "date" { $results[$name]=UnBodge-Date $tmp.trim() }
                            "counts" { $results[$name]=Unpack-Params $tmp }
                            "string" { $results[$name] = """$($tmp.trim())""" }     
                            default { $results[$name] = $tmp.trim() }       
                        }
                    }
                    #-- convert bytes statistics to numbers
                    if ($name -eq "Bytes" -and ($results[$name] -ne $null)) {
                        $tmp=@()     
                        $results[$name].split($Delim) | % {
                            #-- convert value to MBytes 
                            $Bytes=$_
                            if ($Bytes -match "m|g|t|k") {
                                switch ($Bytes.split(" ")[1]) {                
                                    "m" {$conv=1MB*$Bytes.replace(" m","MB")/1MB}
                                    "k" {$conv=1KB*$Bytes.replace(" k","KB")/1KB}
                                    "g" {$conv=1GB*$Bytes.replace(" g","GB")/1GB}
                                    "t" {$conv=1TB*$Bytes.replace(" t","TB")/1TB}
                                }
                            } else { 
                                #-- copy original value, no unit sign detected
                                $conv = $Bytes
                            }
                            #-- convert size 
                            switch ($UnitSize) {
                                "MB" {$tmp+=($conv/1MB)}
                                "KB" {$tmp+=($conv/1KB)}
                                "GB" {$tmp+=($conv/1GB)}
                                "TB" {$tmp+=($conv/1TB)}
                                default {$tmp+=($conv)} #-- no conversion needed. Size is in Bytes
                            }              
                        }
                        #-- rebuild string
                        $results[$name]=$tmp -join $delim                    
                    } #-- end of converting bytes statistics          
                } #-- end of parsing headerparam fields


                if ( $fp ) {
                    #-- parse the complete file
                    write-host "Parsing $($reader.BaseStream.Length) bytes" -NoNewLine

                    # Now go through the file line by line
                    $FT_counters=@{}
                    $DT_counters=@{}
                    $FileTags.GetEnumerator() | select name | %{ $FT_counters.add($_.name.split("|")[1],0)}
                    $DirTags.GetEnumerator() | select name | %{ $DT_counters.add($_.name.split("|")[1],0)}
                    $reader.BaseStream.Position=0
                    $filesdone = $false
                    $linenumber=0
                    $FileResults=@{}
                    $newest=[datetime]"1/1/1900"
                    $oldest=Get-Date
                    $linecount++
                    $firsttick=$elapsedtime.elapsed.TotalSeconds
                    $tick=$firsttick+$refreshrate
                    $LastLineLength=1

                    try {
                        do {
                            $line = $reader.ReadLine()
                            $linenumber++
                            if (($line -eq "-------------------------------------------------------------------------------" -and $linenumber -gt 16)  ) { 
                                # line is end of job
                                $filesdone=$true
                            } elseif ($linenumber -gt 16 -and $line -gt "" ) {
                                #-- split line according to TAB
                                $buckets=$line.split($tab)

                                if ( $buckets.count -gt 3 ) {
                                    #-- line contains file information
                                    $status=$buckets[1].trim()
                                    $FileResults["$status"]++
                                    #-- File tag counters
                                    if ($status -ceq "Newer") { $FT_counters["$status" +" XN"]++ }
                                    elseif ($status -ceq "Older") {  $FT_counters["$status" +" XO"]++ } 
                                    elseif ($status -ceq "Changed") {  $FT_counters["$status" +" XC"]++ }
                                    elseif ($status -ceq "same") {  $FT_counters["$status" +" IS"]++ }
                                    else {$FT_counters["$status"]++}

                                    #-- Get Timestamp from file
                                    $SizeDateTime=$buckets[3].trim()
                                    if ($sizedatetime.length -gt 19 ) {
                                        $DateTime = $sizedatetime.substring($sizedatetime.length -19)
                                        if ( $DateTime -as [DateTime] ){
                                            $DateTimeValue=[datetime]$DateTime
                                            if ( $DateTimeValue -gt $newest ) { $newest = $DateTimeValue }
                                            if ( $DateTimeValue -lt $oldest ) { $oldest = $DateTimeValue }
                                        }
                                    }
                                } elseif ($buckets.count -eq 3) {
                                    #-- line contains directory information
                                    #-- Directory tag counters
                                    $status=$buckets[1].Substring(0,10).trim()
                                    if ($status.length -gt 0){
                                        $DT_counters["$status"]++
                                    } else {
                                        $DT_counters["Exist"]++
                                    }
                                }
                            }

                            #-- progress indicator 
                            if ( $elapsedtime.elapsed.TotalSeconds -gt $tick ) {
                                $line=$line.Trim()
                                if ( $line.Length -gt 48 ) {
                                    $line="[...]"+$line.substring($line.Length-48)
                                }
                                $line="$([char]13)Parsing > $($linenumber) ($(($reader.BaseStream.Position/$reader.BaseStream.length).tostring("P1"))) - $line"
                                write-host $line.PadRight($LastLineLength) -NoNewLine
                                $LastLineLength = $line.length
                                $tick=$tick+$refreshrate                        
                            }

                        } until ($filesdone -or $reader.endofstream)
                    }
                    finally {
                        $reader.Close()
                    }

                    $line=$($([string][char]13)).padright($lastlinelength)+$([char]13)
                    write-host $line -NoNewLine
                }

                #-- write results
                $writer.Write("`"$file`"")
                $line="`"$file`""
                #-- write values
                foreach ( $HeaderParam in $HeaderParams.GetEnumerator() | Sort-Object Name ) {
                    $name = "$(Remove-Key $HeaderParam.Name)"
                    if ( $results[$name] ) {
                        $writer.Write("$Delim$($results[$name])")
                        $line=$line+"$Delim$($results[$name])"
                    } else {
                        if ( $ErrorFooter ) {
                            #-- placeholder
                        } elseif ( $HeaderParam.Value -eq "counts" ) {
                            #-- write summary counters
                            $writer.Write($Delim+$Delim+$Delim+$Delim+$Delim+$Delim)
                            $line=$line+"$Delim$($results[$name])"
                        } else {
                            $writer.Write($Delim) 
                            $line=$line+"$Delim$($results[$name])"
                        }
                    }
                }

                if ( $ErrorFooter ) {
                    $tmp = $($ErrorFooter -join "").substring(20)
                    $tmp=$tmp.substring(0,$tmp.indexof(")")+1)+$Delim+$tmp
                    $writer.write($delim+$delim+"$tmp")
                    $line=$line+"$Delim$($results[$name])"
                } elseif ( $fp ) {
                    #-- write values from file parsing      
                    $writer.write($delim+"$LineCount"+$delim+"$($newest.ToString('dd/MM/yyyy hh:mm:ss'))"+$delim+"$($oldest.ToString('dd/MM/yyyy hh:mm:ss'))")      
                    $line=$line+"$Delim$($results[$name])"
                    #-- write File tag counters 
                    foreach ($fileTag in $filetags.GetEnumerator() | Sort-Object Name ) {
                    $writer.write($delim+"$($FT_counters[$Filetag.name.split("|")[1]])")
                    $line=$line+$delim+$($FT_counters[$Filetag.name.split("|")[1]])
                    }
                    #-- write directory tag counters
                    foreach ($DirTag in $DirTags.GetEnumerator() | Sort-Object Name ) {
                    $writer.write($delim+"$($DT_counters[$dirtag.name.split("|")[1]])")
                    $line=$line+$delim+$($dT_counters[$DirTag.name.split("|")[1]])
                    }
                }
                #-- EOL
                $writer.WriteLine()
                $table=$table+"`n"+$line
            } else {
                #-- file is not a RoboCopy log file
                #write-host -foregroundcolor darkgray "$($file.name) is not recognised as a RoboCopy log file"
                write-warning "$($file.name) is not recognised as a RoboCopy log file"
            }
        }
    } #- -end of Process

    End{
        #-- write and output results
        write-host "$filecount files scanned in $($elapsedtime.elapsed.tostring()), $($ProcessCounts["Processed"]) complete, $($ProcessCounts["Error"]) have errors, $($ProcessCounts["Incomplete"]) incomplete"
        write-host  "Results written to $($writer.basestream.name)"
        $CSVFile=$writer.basestream.name
        $writer.close() #-- yes, we are done writing
        #-- create output object, containing name of CSV file and Report object
        $Output=New-Object -TypeName psobject -Property @{ReportFileName=$CSVFile
                                                          Report=ConvertFrom-Csv -Delimiter $Delim -InputObject $table
                                                          Summary=@()}

        #-- summize, build a Robocopy like Summary
        $Types=@("Dirs","Files",("Bytes".replace("B",$SizeUnit)),"Times","Speed")
        $types | % {
            $row= "" | select  Type,Total,Copied,Skipped,Mismatch,FAILED,Extras
            $row.type=$_
            if ($row.type -ilike "Times") { 
                #-- calculate times
                $Output.Report | %{
                    if ($_.ended.length -gt 0){
                        #-- only use data from complete RC logs
                        $row.total=$row.total+[timespan]$_."Times Total"
                        $row.Failed=$row.Failed+[timespan]$_."Times Failed"
                        $row.Copied=$row.Copied+[timespan]$_."Times Copied"
                        $row.Extras=$row.Extras+[timespan]$_."Times Extras"   
                    } else {
                        Write-Verbose ("RC log "+$_.file+" skipped for summary calculation. Log file is not complete.")
                    }     
                }
            } elseif ($row.type -ilike "Speed") {
                #-- Calculate speed
                $Duration=0
                $output.Report | %{
                        if ($_.ended.length -gt 0){
                            #-- only use data from complete RC logs
                            $Duration=$Duration+[timespan]$_."Times Copied"
                        }
                    }
                $row.Copied="{0:N1}" -F ((($output.report | Measure-Object -Property (("Bytes".replace("B",$SizeUnit))+" Copied") -Sum).sum) / $Duration.seconds)
            } else {
                #-calculate counters

                Write-Verbose ("%%%%%%%%%%%%%%%%%%%%"+$row.type)

                $row.Total="{0:N1}" -f ($output.report | Measure-Object -Property ($row.type+" Total") -Sum).sum
                $row.Copied="{0:N1}" -f($output.report | Measure-Object -Property ($row.type+" Copied") -Sum).sum
                $row.Skipped="{0:N1}" -f($output.report | Measure-Object -Property ($row.type+" Skipped") -Sum).sum
                $row.Mismatch="{0:N1}" -f($output.report | Measure-Object -Property ($row.type+" Mismatch") -Sum).sum
                $row.Failed="{0:N1}" -f($output.report | Measure-Object -Property ($row.type+" Failed") -Sum).sum
                $row.Extras="{0:N1}" -f($output.report | Measure-Object -Property ($row.type+" Extras") -Sum).sum
            }
            $Output.Summary+=$row
        }
        [System.Threading.Thread]::CurrentThread.CurrentCulture = $OldCulture
        Write-Verbose ("Changed Locale back to "+$oldCulture.Name)
        return($output) #-- done, fini, finaly....
    } #-- end of End
}

Analyse-RC_Log -SourcePath C:\Desktop\ServerLog.txt -ExcelCSV -fp -unitsize

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows                              
-------------------------------------------------------------------------------

  Started : Wed Nov 15 21:05:18 2015

   Source : E:\DATA\SomeStagingDir\
     Dest : \\DestinationServer\\SomeStagingDir\SomeDir\

    Files : *.zip

  Options : /FFT /V /S /E /COPY:DAT /Z /MAXAGE:5 /IPG:100 /R:0 /W:1 

------------------------------------------------------------------------------

                       0    E:\DATA\SomeStagingDir\
    *EXTRA Dir        -1    \\DestinationServer\\SomeStagingDir\SomeDir\Lxx\
    *EXTRA Dir        -1    \\DestinationServer\\SomeStagingDir\SomeDir\New folder\
    *EXTRA Dir        -1    \\DestinationServer\\SomeStagingDir\SomeDir\nxx1\
                       1    E:\DATA\SomeStagingDir\Server_1\
        New File           1.8 m    Server_1.log.zip
2015/11/15 21:05:18 ERROR 112 (0x00000070) Copying File E:\DATA\SomeStagingDir\Server_1\Server_1.log.zip
There is not enough space on the disk.



------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         2         0         2         0         0         3
   Files :         1         0         0         0         1         0
   Bytes :    1.82 m         0         0         0    1.82 m         0
   Times :   0:00:00   0:00:00                       0:00:00   0:00:00

   Ended : Wed Nov 25 20:05:18 2015

Thank You..

SOAPUI MockServiceRunner on Linux without X Config

2019-01-10T16:17:00+00:00

Updated

After I struggled for this problem, I am going to keep it for my refrence. As the title says, We will see in the post how to run a SoapUI MockService on a (X)nix box without needing to start the X Server Configuration .

SOAPUI - Installing on Linux/Unix

Source:http://www.soapui.org/Getting-Started/installing-on-linuxunix.html

64-bit systems - prerequisite

Make sure that you have Java (JRE) installed on your system

Installation

Download the Linux binary zip (no JRE) from http://www.soapui.org/
Unzip it into a preferable directory such as your home folder or /opt
Make sure that you have proper permissions on the unziped soapUI folder
Go into the folder and run $bin/soapui.sh (open source) or $bin/soapui-pro.sh (Pro)

Mock Services

Source: http://www.soapui.org/Test-Automation/mock-services.html

Running your MockServices from the command-line is equally simple; use the bundled mockrunner.bat/.sh file with the following arguments:

   m : The name of the MockService to run  
   p : The local port to listen on, overrides the port configured for the MockService  
   a : The local path to listen on, overrides the path configured for the MockService  
   b : Turns off blocking when mockRunner has been started, which is required when wanting to run the MockServiceRunner with (for example) nohup or as a Windows Service  
   s : The soapui-settings.xml file to use  
   x : Sets project password for decryption if project is encrypted  
   v : Sets password for soapui-settings.xml file   
   D : Sets system property with name=value  
   G : Sets global property with name=value  
   P : Sets project property with name=value  
   S : Saves the project after running the mockService(s)  
   f : Sets the output folder to export results to ( soapUI Pro only )  
   o : Opens the Coverage Report in a browser (with the -g option) ( soapUI Pro only )  
   g : Sets the output to include Coverage HTML reports ( soapUI Pro only )

The distribution contains a mockservicerunner.bat script for running MockServices in the bin directory, for example;

mockservicerunner.bat -m"IOrderService MockService" "C:\demo2-soapui-project.xml"

Runs the specified MockService as follows:

Which can now be invoked from soapUI or any other client. Terminate the runner by pressing the return key in the console, which will shutdown as follows:

MockServiceRunner without X configuration

Source:http://www.soapui.org/forum/viewtopic.php?t=1023

When trying to use mockservicerunner.sh in Linux/Solaris, you may get an error because of the X windows configuration.

If you want to know is how to use the mockservicerunner.sh in a manner that it does not need an X configuration. After all, why does mockservicerunner.sh need an X configuration ? The main reason for executing it in a command line is to avoid that.

If you find any of the errors above adding the java.awt.headless

i.e

mockservicerunner.sh ../soapui-project.xml -Djava.awt.headless=true

Use xming, xshell to display linux gui to windows desktop (x11 forwarding)

Source: http://www.doxer.org/learn-linux/use-xming-xshell-to-display-linux-gui-to-windows-desktop-x11-forwarding/

Download xming, install it on your windows pc system. You can go to http://sourceforge.net/projects/xming/files/ to download.

After installation xming on your windows, log in linux/solaris server 192.168.0.3. Set environment variable DISPLAY to the ip address of your windows, and append a :0 to it:

export DISPLAY=192.168.0.4:0

Then you must allow X11 forwarding in sshd configuration file. That is, set X11Forwarding to yes in /etc/ssh/sshd_config and restart your sshd daemon.

After this, you need set 192.168.0.3(linux/solaris) to the allowed server list on your windows. Edit X0.hosts which locates at the installation directory of xming(For example, C:\Program Files\Xming\X0.hosts), add a new entry in it:192.168.0.3, the ip address of linux/solaris that you want to run x11 utility from.

Then, restart xming on your windows. And on solaris/linux server(192.168.0.3), run a X11 programe, like

/usr/openwin/bin/xclock &

You will then see a clock gui pop up in your windows pc.

Voila, You are done.

"All credits & kudos to the concerned websites and the users who have documented these resolutions."

AWS Active Directory with OpenVPN

2018-12-28T00:00:00+00:00

Many wonder what the benefits of Active Directory are in the modern era. We can leverage Active Directory to control access to resources. One absolutely great feature is Group Policy Objects (GPOs). GPOs enables seamless monitoring of Windows machines with Policies like OS updates, screen lock, and more.

I wanted to document some setup required for AWS MS Active Directory - Aka. Directory Services and OpenVPN

Prerequistites

Launch Active Directory
Setup you own EC2 instance and install Open VPN, or Launch OpenVPN Access Server from AWS Market place
You will need an EC2 instance to manage the Active Directory objects, See further how to Install the Active Directory Administration Tools on Windows Server 2016

Target Architecture

Active Directory Management

Manage Users and Groups in AWS Managed Microsoft AD
Create User

Manually Join EC2 Instance to the AD

To manually join an existing Amazon EC2 Windows instance to a Simple AD or AWS Directory Service for Microsoft Active Directory directory, the instance must be launched as specified in Seamlessly Join a Windows EC2 Instance

To join a Windows instance to a Simple AD or AWS Managed Microsoft AD directory
Connect to the instance using any Remote Desktop Protocol client.
Open the TCP/IPv4 properties dialog box on the instance.
Open Network Connections usisng command.
%SystemRoot%\system32\control.exe ncpa.cpl
Change the DNS server addresses, change the Preferred DNS server and Alternate DNS server addresses to the IP addresses of the AWS Directory Service-provided DNS servers.

Open the System Properties dialog box for the instance, select the Computer Name tab, and choose Change.
Use command %SystemRoot%\system32\control.exe sysdm.cpl

In the Member of field, select Domain, enter the fully-qualified name of your AWS Directory Service directory, and choose OK.
When prompted for the name and password for the domain administrator, enter the username and password of an account that has domain join privileges.
For more information about delegating these privileges, see Delegate Directory Join Privileges for AWS Managed Microsoft AD.

Note:
You can enter either the fully-qualified name of your domain or the NetBios name, followed by a backslash (), and then the user name, in this case, Admin. For example, corp.example.com\Admin or corp\Admin.

Restart the instance to have the changes take effect.

To allow domain users RDP access to the domain joined Windows instances, follow these steps:

You will get error "The connection was denied because the user account is not authorized for remote login."
To solve it follow this:
Connect to your Windows EC2 instance using RDP.
Create a user. Repeat this step if you need more than one user.
Create a security group.
Add the new users to the new security group.
Open Group Policy Management. Select your domain.s Forest, expand Domains, and then expand your domain name.
Expand your delegated OU (NetBIOS name of the directory). Open the context (right-click) menu for Computers, and then choose Create a GPO in this domain, and Link it here.
For Name, enter a name, and then choose Ok.
In the navigation pane, expand Computers. Open the context (right-click) menu for the policy, and then choose Edit.
In the Computer Configuration section of the navigation pane, expand Preferences, Control Panel Settings.
Open the context (right-click) menu for Local Users and Groups, and then choose New, Local Group.
For Group name, choose Remote Desktop Users (built-in), and then choose Add.
For Name, enter the name of the security group that you created in step 3, and then choose Ok.
This policy updates your environment at the next policy refresh interval. To force the policy to apply immediately, run the gpupdate /force command on the target server.

Install the Active Directory Administration Tools on Windows Server 2016

Open Server Manager from the Start screen by choosing Server Manager.
In the Server Manager Dashboard, choose Add roles and features,
In the Add Roles and Features Wizard choose Installation Type, select Role-based or feature-based installation, and choose Next.
Under Server Selection, make sure the local server is selected, and choose Features in the left navigation pane.
In the Features tree, open Remote Server Administration Tools, Role Administration Tools, select AD DS and AD LDS Tools, scroll down and select DNS Server Tools, and then choose Next.
Review the information and choose Install. When the feature installation is finished, the Active Directory tools are available on the Start screen in the Administrative Tools folder.

Manage GPO permissions

Open VPN

Authenticate VPN using LDAP
Authenticate VPN using LDAP - Another article

Get More Out Of Google

2018-08-12T21:34:00+01:00

Updated the links.

Once I stumbled upon an infographics from HackCollege
So sharing it here.
Learn the tricks and Enjoy. If you need a printable pdf version, head over here -- Thanks to Brett -- Thanks to Google

Search string faster with Ruby

2018-07-20T15:59:00+01:00

Is grep Slow? Sure it is. Read on.. Just a few days back there was an electrical outage and lot of applications were dead.
In one case, lot of customer orders could not be processed. Hence, there rose a need of manual intervention and extraction of orders (XML) from a logfile and re-feeding them to another system. The task was simple, I had order numbers in orders.txt and I had to write a shell script to grep for a particular xml containing each of these orders, extract XML and create a file for each order.

Shell script to loop and find an order in another file

cat extract.sh
--------------
i=0
echo "Extraction Script Started at:" `date`
while read order_id; do
  filename=$order_id"_tmp.xml"
  finalfilename=$order_id".xml"
  grep ".*$order_id." *.log > $filename && echo "written xml for $order_id in $filename" || echo $order_id >> Orders_not_found.txt
  cat $filename | sed -e 's|text-to-remove|new-text|g' | sed -e 's|\*\*\*||g' > $finalfilename
  rm $filename
  ((i++))
done < orders.txt
echo "Extraction Script Ended at:" `date`

But the problem was that the log file in which I was searching was too huge. It was 5GBs in total. Hence the grep was taking minimum 4-5 Minutes to search one order and create an xml file for that. Clearly this was not a solution, as I had to find thousand orders in those log files and it was very critical for end customer.

If my calulation was right, I had to spend:
4 Mins = 1 Xml
60 Mins = 1 Hr = 15 Xml
at this rate I would have spent atleast 3-4 days CPU time , to get all those 1000 XMLs. (Not to mention the pain of getting screwed and frustration). Meaning which, we all would have been screwed over and over again for 3-4 days by the customer.

Enter Ruby: One liner saved us.
I used this ruby command, to first find the relevant generic string then create order xml files using a normal shell script as above. I thought I would keep this fir future reference.

$ ruby -pe 'next unless $_ =~ /.*Number.*/' < 5GB-file.log >> 6Mb-file-reduced.log

Wondereful, Ruby took just few minutes to grep the regular exp string into a 5GB log file, and now I had to search orders into this smaller reduced size intermediate file.

Thus, this saved us 3 days and did wonders in just half an hour.

Voila !!!

Credit for the one liner goes to Garry Tan, where I found this wonderful ruby command.

How to use Angular-Datatables

2018-07-03T10:47:00+01:00

This post is about quick demo of angularjs + datatables usage. I am not gonna write too much theory, So here I will give only code snippet below.

References:

http://l-lin.github.io/angular-datatables
https://datatables.net

Code:

index.html

somejavascript.js

myModule.controller('myDatatablesIntCtrl', function($scope, $rootScope, $http, DTOptionsBuilder, DTColumnBuilder, DTDefaultOptions) {

var vm = this;
vm.dtOptions = DTOptionsBuilder.newOptions()
    .withDOM('<"row"<"col-md-8 col-sm-12"<"inline-controls"l>><"col-md-4 col-sm-12"<"pull-right"f>B>>t<"row"<"col-md-4 col-sm-12"<"inline-controls"T>><"col-md-4 col-sm-12"<"inline-controls text-center"i>><"col-md-4 col-sm-12"p>>')
//.withDOM('Blfrtip')
.withDisplayLength(25)
//.withScroller()
//.withOption('deferRender', true)
//.withOption('scrollY', 200)
//.withOption('scrollX', '100%')
//.withOption('responsive', true)
//.withColVis()
//.withOption('order', [[3, 'desc']])
// Add Bootstrap compatibility
.withBootstrap()
.withBootstrapOptions({
        TableTools: {
            classes: {
                container: 'btn-group',
                buttons: {
                    normal: 'btn btn-lg btn-primary'
                }
            }
        },
        ColVis: {
            classes: {
                masterButton: 'btn btn-primary'
            }
        },
        pagination: {
            classes: {
                ul: 'pagination pagination-sm'
            }
        }

    })
.withOption('retrieve', true)

//.withButtons(['columnsToggle', {
//    extend: 'collection',
//    text: 'Hide columns',
//    buttons: ['columnsVisibility'],
//    visibility: false
//}])

// Add Table tools compatibility
.withTableTools('/plugins/datatables/copy_csv_xls_pdf.swf')
    .withTableToolsOption('sRowSelect', 'multi')
    .withTableToolsButtons([{
        'sExtends': 'copy',
        'sButtonText': 'Copy To Clipboard',
        'bSelectedOnly': true,
        'oSelectorOpts': {
            filter: 'applied',
            order: 'current'
        }
    }, {
        'sExtends': 'collection',
        'sButtonText': 'Export Data',
        'aButtons': [{
                'sExtends': 'xls',
                'sButtonText': 'XLS',
                'sFileName': '.xls',
                'bSelectedOnly': true,
                'oSelectorOpts': {
                    filter: 'applied',
                    order: 'current'
                }
                // 'fnMouseover': function ( nButton, oConfig, oFlash ) {
                //     alert( 'Mosue over' );
                // }
            }, {
                'sExtends': 'pdf',
                'bFooter': true,
                'bHeader': true,
                // 'mColumns': [0, 1, 2, 3, 4, 5, 6, 7],
                'sPdfOrientation': 'landscape',
                'sFileName': '.pdf',
                'bSelectedOnly': true,
                'oSelectorOpts': {
                    filter: 'applied',
                    order: 'current'
                }
            }, {
                'sExtends': 'csv',
                'sButtonText': 'CSV',
                'sFileName': '.csv',
                'bSelectedOnly': true,
                'oSelectorOpts': {
                    filter: 'applied',
                    order: 'current'
                }
            }
            // {
            //     'sExtends': 'text',
            //     'sButtonText': 'PNG',
            //     'bHeader': false,
            //     'bSelectedOnly': true,
            //     'oSelectorOpts': {
            //         'page': 'current'
            //     },
            //     'fnClick': function() {
            //         // Convert html table to canvas element
            //         html2canvas($(".active table"), {
            //             onrendered: function(canvas) {
            //                 var context = canvas.getContext("2d");
            //                 // Save canvas to file
            //                 canvas.toBlob(function(blob) {
            //                     saveAs(blob, '.png');
            //                 }); 
            //             }
            //         });
            //     }
            // }
        ]

    }]);

$scope.$on("InvokeDatatablesCtrl", function(event, args) {

    console.log('Table initialisation start: ' + new Date().getTime());

    var table = $('#mydatatable_id').DataTable();
    $('#mydatatable_id')
        .on('init.dt', function() {
            console.log('Table initialisation complete: ' + new Date().getTime());
            table.buttons().container().appendTo('#dt-buttons');
        }).dataTable();



    // console.log(args);
    // console.log("Inside myDatatablesIntCtrl");
    // $rootScope.alert = "Please wait...";
    //              $rootScope.error = "";
    // $.ajax({
    //  url : args.url,
    //  data : {},
    //  successFunction : function(data,status,headers,config)
    //  {

    //      $scope.tabs.datatableslist = data.data;  

    //  },
    // });

    // $resource(args.url).get().$promise.then(function(data)
    //      {

    //          if('code' in data && data.code == "0")
    //          {
    //             $rootScope.alert = data.message;
    //             $rootScope.error = "";
    //              if('redirect' in data && data.redirect != "")
    //              {
    //                  window.location = data.redirect;
    //              }

    //             $scope.tabs.viewrota.list = data.data;
    //             //console.log($scope.tabs.viewrota.list);

    //         }
    //          else if ('error' in data && data.error != "")
    //          {
    //                  $rootScope.error = data.error;  
    //          }

    // });
});

});

html or ctp file

Heading1	Heading2	Heading3
Column1	Column2	Column3

How to use create ajax dropdown from database models in Rails

2018-05-02T20:17:00+01:00

This post is about quick demo of Ajax based dropdown in rails. The dropdown fetches the data from a collection in database.

index.html.erb

<%= f.collection_select :vertical, @verticals, :vertical, :vertical, {:prompt => "Select a Vertical"}, {:class => "dropdown_vertical btn btn-default dropdown-toggle"} %>

<%= f.collection_select :subVertical, @subVerticals, :subVertical, :subVertical, {:prompt => "Select a Sub Vertical"}, {:class => "dropdown_subVertical btn btn-default dropdown-toggle"} %>

custom.js

$("#search_vertical").on('change', function(){
var listitems = [];

$.ajax({
      url: "populate_subVerticals",
      type: "GET",
      data: {vertical_name: $(this).val()},
      success: function(data) {
      $("#search_subVertical").children().remove();
      $("#search_subVertical").append('' + "Select a Sub Vertical" + '');
      $.each(data,function(key, value) 
      {
      listitems += '' + value.subVertical + '';
      });

      $("#search_subVertical").append(listitems);

      }
  })
});

Controller

def populate_subVerticals
    vertical_name = params[:vertical_name]
    @verticals = Vertical.where(:vertical => vertical_name).select("id","subVertical").all
    respond_to do |format|
      format.json { render json: @verticals }
    end
  end

routes.rb file

get 'populate_subVerticals', to: 'verticals#populate_subVerticals'

That's it folks.