Managing cloud infrastructure across private and public clouds can be overwhelming. Cloud Resources Map solves this challenge by offering a read-only observability platform, empowering teams to monitor resources securely without risking accidental changes or deletions. Key Features 🌐 Unified Resource View Consolidates resources from major providers—AWS, Azure, GCP, Aliyun, and Xen Orchestra—into a single pane of glass. 📍 Interactive Mapping Visualizes resources geographically on a dynamic map with clickable regions for detailed insights. 🎨 Smart Region Recommendations Uses advanced analytics to recommend optimal deployment regions, clearly marked with distinct colors and pins. 📊 Insightful Analytics Interface Offers comprehensive charts and graphs for actionable insights into resource utilization and infrastructure health. ⚠️ Proactive Alerting Provides real-time webhook notifications for key events like unusual CPU/memory usage or unexpected deletions. 🔍 Advanced Filters Quickly locates specific resources within complex, multi-cloud environments with robust search capabilities. 💡 Secure, Read-Only Observability Built exclusively for monitoring, ensuring maximum security and peace of mind. Why It Matters Cloud Resources Map empowers SysOps teams and cloud administrators to achieve: • Enhanced Visibility: A unified view of all resources. • Increased Efficiency: Streamlined operations and reduced management complexity. • Informed Decisions: Actionable analytics for strategic planning. By providing a secure, interactive, and scalable interface, this app addresses the core challenges of modern cloud management—visibility, security, and operational efficiency.

Overview: This project involved designing and deploying a secure, scalable, and highly available architecture using AWS services for managing containerized workloads, databases, and monitoring tools. The infrastructure was provisioned using AWS CloudFormation templates, and AWS Secrets Manager was utilized for secure management of sensitive data.

---

Key Responsibilities and Contributions:

• Designed a cloud architecture incorporating AWS EKS, RDS, ALB, ELK Stack, and supporting services.

• Automated the provisioning of infrastructure components using AWS CloudFormation.

• Integrated AWS Secrets Manager for secure storage and retrieval of sensitive information.

• Implemented centralized logging and monitoring using ELK Stack.

• Ensured scalability and high availability of applications with ALB and EKS.

• Configured Amazon RDS (Oracle and PostgreSQL) for database management.

• Secured access to the VPC using an EC2 bastion host.

• Deployed static assets and application backups to Amazon S3.

• Utilized Amazon ECR for hosting Docker container images.

---

Architecture Components

Cloud

• Provides a highly available and scalable environment for hosting cloud resources and applications.

VPC (Virtual Private Cloud)

• A secure, isolated network hosting the core resources of the architecture.

Key Components:

1. AWS ALB (Application Load Balancer): Distributes incoming traffic to EKS.

2. EKS Cluster: Orchestrates containerized workloads.

3. Cluster ELK (Elasticsearch, Logstash, Kibana): Manages logging and monitoring.

4. Amazon RDS: Provides databases (Oracle and PostgreSQL) for application use.

5. Amazon S3: Stores static assets and application backups.

6. Amazon ECR: Hosts Docker images for application deployments.

7. EC2 Bastion Host: Provides secure access to the VPC.

8. AWS Secrets Manager: Ensures secure handling of sensitive information like database credentials.

---

Technology Stack:

• Infrastructure Management: AWS CloudFormation

• Container Orchestration: Amazon EKS

• Database Management: Amazon RDS (Oracle and PostgreSQL)

• Monitoring and Logging: ELK Stack

• Secrets Management: AWS Secrets Manager

• Storage: Amazon S3

• Container Registry: Amazon ECR

• Access Management: EC2 Bastion Host

---

Challenges Faced:

1. Scaling for High Traffic:

   • Implemented ALB and auto-scaling features in EKS to handle variable traffic loads effectively.

2. Secure Management of Sensitive Data:

   • Integrated AWS Secrets Manager to ensure credentials were securely stored and rotated automatically.

3. Monitoring and Troubleshooting:

   • Set up ELK Stack to centralize logs and enable real-time monitoring of system performance.

---

Results Achieved:

• Improved scalability and fault tolerance for workloads hosted on EKS.

• Enhanced security by isolating the network within a VPC and managing secrets securely.

• Streamlined infrastructure deployment through CloudFormation templates, reducing setup time by 40%.

• Enabled comprehensive monitoring with the ELK Stack, resulting in faster issue detection and resolution.

---

Takeaway:

This project showcases my ability to design and implement robust cloud architectures, automate infrastructure provisioning, and integrate advanced AWS services to achieve scalability, security, and high availability. It demonstrates my expertise in AWS CloudFormation, EKS, ELK Stack, RDS, and Secrets Manager.

Prerequisites

Before proceeding, make sure you have the following prerequisites:

1. Kubernetes cluster up and running.
2. kubectl command-line tool installed and configured to access your Kubernetes cluster.
3. Basic knowledge of Kubernetes concepts and YAML configuration files.

Step 1: create a namespace

To get started, create a namespace inside k8s Cluster, to do that you can simply run the following command:

kubectl create ns demo-elastic

Step 2: create persistent volume For Data-elastic

1. Create a file called elasticsearch-pvc.yml and add the following YAML configuration:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elasticsearch-pvc
  labels:
    component: elasticsearch
spec:
  # storageClassName: standard
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1G
---

2. Apply the configuration to deploy PVC:

kubectl apply -f elasticsearch-pvc.yaml -n demo-elastic

Step 3: Deploy Elasticsearch

1. Create a file called elasticsearch-deployment.yaml and add the following YAML configuration:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      component: elasticsearch
  template:
    metadata:
      labels:
        component: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: single-node
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        volumeMounts:
        - name: elasticsearch-config
          mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          subPath: elasticsearch.yml
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/data
        resources:
          limits:
            cpu: 500m
            memory: 4Gi
          requests:
            cpu: 500m
            memory: 4Gi
      # Allow non-root user to access PersistentVolume
      securityContext:
        fsGroup: 1000
      restartPolicy: Always
      volumes:
      - name: elasticsearch-config
        configMap:
          name: elasticsearch-config
      - name: elasticsearch-data
        persistentVolumeClaim:
          claimName: elasticsearch-pvc

1.2 Apply the configuration to deploy Elasticsearch:

kubectl apply -f elasticsearch-deployment.yaml -n demo-elastic

1.3 Create a file called elasticsearch-service.yaml and add the following YAML configuration:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    component: elasticsearch
spec:
  type: NodePort
  selector:
    component: elasticsearch
  ports:
  - port: 9200
    targetPort: 9200

1.4 Apply the configuration to service Elasticsearch:

kubectl apply -f elasticsearch-service.yaml -n demo-elastic

1.5 Create a file called elasticsearch-configmap.yaml and add the following YAML configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-config
  labels:
    component: elasticsearch
data:
  elasticsearch.yml: |
    cluster.name: Demo-Elastic
    node.name: Master
    path.data: /usr/share/elasticsearch/data
    http:
      host: 0.0.0.0
      port: 9200
    bootstrap.memory_lock: true
    transport.host: 127.0.0.1
    xpack.license.self_generated.type: basic
    # Enable xpack.security which is provided in basic subscription
    xpack.security.enabled: true
    # Disable unused xpack features 
    xpack.monitoring.enabled: false
    xpack.graph.enabled: false
    xpack.watcher.enabled: false
    xpack.ml.enabled: false

With the following command, we can now set up passwords for built-in users and install Elasticsearch to our cluster. During kibana and fluentd configuration, we’ll employ “elastic” users.

Disclaimer: I recommend enhancing the use of credentials in your production deployment by using Secrets over Configmaps, using distinct users for Kibana, and becoming more familiar with the least access concept.

$ kubectl exec -it elasticsearch-0 -n demo-elastic -- bin/elasticsearch-setup-passwords auto -b

...
##Changed password for 
user elasticPASSWORD elastic = ismail-wajdi-7415b3176

1. 6 Apply the configuration to config-Map Elasticsearch:

kubectl apply -f elasticsearch-configmap.yam -n demo-elastic

Step 4: Deploy Kibana

1.1 Create a file called elasticsearch-kibana.yaml and add the following YAML configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
spec:
  selector:
    matchLabels:
      component: kibana
  template:
    metadata:
      labels:
        component: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.12.0
        env:
        - name: ELASTICSEARCH_URL
          value: http://elasticsearch:9200
        - name: XPACK_SECURITY_ENABLED
          value: "true"
        - name: ELASTICSEARCH_USERNAME
          valueFrom:
            configMapKeyRef:
              name: kibana-config
              key: elasticsearch_username
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            configMapKeyRef:
              name: kibana-config
              key: elasticsearch_password
        envFrom:
        - configMapRef:
            name: kibana-config
        ports:
        - containerPort: 5601
          name: http
          protocol: TCP

1.2 Apply the configuration to deploy Kibana:

kubectl apply -f elasticsearch-Kibana.yaml -n demo-elastic

1.3 Create a file called Kibana-service.yaml and add the following YAML configuration:

apiVersion: v1
kind: Service
metadata:
  name: kibana
  labels:
    component: kibana
spec:
  type: NodePort
  selector:
    component: kibana
  ports:
  - port: 5601
    targetPort: 5601

1.4 Apply the configuration to service Kibana:

kubectl apply -f Kibana-service.yaml -n demo-elastic

1.5 Create a file called Kibana-configmap.yaml and add the following YAML configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-config
  labels:
    component: kibana
data:
  elasticsearch_username: elastic
  elasticsearch_password: ismail-wajdi-7415b3176

1.6 Apply the configuration to config-Map Kibana:

kubectl apply -f Kibana-configmap.yaml -n demo-elastic

Step 5: Accessing Elasticsearch and Kibana

Before proceeding, make sure you have the following prerequisites:

1. Obtain the external IP addresses and Port of the Elasticsearch and Kibana services:

kubectl get svc -n
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
elasticsearch                 NodePort    10.233.39.51    <none>        9200:31012/TCP   6h12m
kibana                        NodePort    10.233.5.197    <none>        5601:32398/TCP   6h4m

2. Once you have the external IP addresses, you can access Elasticsearch and Kibana using a web browser or tools like cURL or a REST client.

3. Accessing Elasticsearch: Open a web browser or use a tool like cURL and visit http://<elasticsearch-external-ip>:9200. If Elasticsearch is running successfully, you will see information about the Elasticsearch cluster, confirming that it is accessible.

4. Accessing Kibana: Open a web browser and navigate to http://<kibana-external-ip>:5601. This will take you to the Kibana login page. Enter the elastic username and the password generated during the setup process. Once authenticated, you will have access to the Kibana dashboard, where you can explore and analyze data, create visualizations and dashboards, and perform various tasks provided by the Kibana interface.

 

By following these steps, you have successfully deployed Elasticsearch with a Kibana instance on Kubernetes, along with password protection. You can now access Elasticsearch to interact with the data and open Kibana to leverage its powerful features for data analysis and visualization. The external IP addresses obtained allow you to access Elasticsearch and Kibana securely from a web browser or through API calls, ensuring that only authorized users can access and utilize these services.