tag:blogger.com,1999:blog-70227555175510073552024-03-23T11:14:04.057+01:00Random thoughts on coding and technologyExploring the Intersection of Innovation and Code: Dive deep into the world of software development with a blog that journeys through the intricacies of .NET, Python, Delphi, and AI. Here, we unravel the complexities of software architecture, offering insights, tips, and thought-provoking discussions. Whether you're a seasoned developer or new to the scene, join us as we navigate the ever-evolving landscape of technology, pushing the boundaries of what's possible in the digital realm.Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.comBlogger516125tag:blogger.com,1999:blog-7022755517551007355.post-79207243811075335562024-02-25T20:46:00.000+01:002024-02-25T20:46:00.693+01:00Running a local LLM on your desktop/server<div style="text-align: left;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg3hDOGQXeH4Y_vcqm6bSZnKS-UqJn69GjQjOntZxMDbUt7Rl3sU6tdbz06B8EI8yJCiS66C_JL4PDadUJxMQCEoOqRXK74UlQU-GlKj6pLk1RivhhKrrFfxOBCViLZ1olrfVJXRCyeSLW28Nf_TAfDkRrKtKok0lMrhXcUhF-MhrKEKrt-tGqXQfBo5Sk" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="" data-original-height="1024" data-original-width="1024" height="240" src="https://blogger.googleusercontent.com/img/a/AVvXsEg3hDOGQXeH4Y_vcqm6bSZnKS-UqJn69GjQjOntZxMDbUt7Rl3sU6tdbz06B8EI8yJCiS66C_JL4PDadUJxMQCEoOqRXK74UlQU-GlKj6pLk1RivhhKrrFfxOBCViLZ1olrfVJXRCyeSLW28Nf_TAfDkRrKtKok0lMrhXcUhF-MhrKEKrt-tGqXQfBo5Sk" width="240" /></a></div><span style="background-color: white;">This article provides guidance on setting up and running a local Large Language Model (LLM) on a desktop environment. The repository contains relevant resources and code, primarily in Jupyter Notebook format, to facilitate the installation and operation of a local LLM. For detailed instructions and code, visit the <a href="https://github.com/JordiCorbilla/Running_Local_LLM" target="_blank">GitHub repository</a>.</span></div><div style="text-align: left;"><span style="background-color: white;"><br /></span></div><div style="text-align: left;"><span style="background-color: white;"><br /></span></div><div style="text-align: left;"><span style="background-color: white;"><br /></span></div><div style="text-align: left;"><span style="background-color: white;"><br /></span></div><div style="text-align: left;"><span style="background-color: white;"><br /></span></div><div style="text-align: left;"><span style="background-color: white;"><br /></span></div><div style="text-align: left;"><span style="background-color: white;"><br /></span></div><div style="text-align: left;"><span style="background-color: white;"><br /></span></div><p><span style="font-size: large;"><span style="background-color: white; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-weight: var(--base-text-weight-semibold, 600);">Download LM Studio:</span></span></p><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><span style="background-color: white;">Go to <a href="https://lmstudio.ai/" rel="nofollow" style="box-sizing: border-box; text-underline-offset: 0.2rem;">https://lmstudio.ai/</a> and download the latest version for Windows:</span></li></ul><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://private-user-images.githubusercontent.com/7347994/302104152-5986f2da-ff44-4a02-9546-1ae866bb39f7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNDE1Mi01OTg2ZjJkYS1mZjQ0LTRhMDItOTU0Ni0xYWU4NjZiYjM5ZjcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YmIzMjVlOGUyN2IxNzM1MTY1OWYxMjJmNzQ3ZTNjODc0YWMzYWVkM2YxYjI3YmUzNmMyNjYzODk0MzBhMzllNyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.R-NICRF-NqmdYowdKXEnsYWaDLFlqsPaJSihWvBJvBg" rel="noopener noreferrer" style="box-sizing: border-box; text-underline-offset: 0.2rem;" target="_blank"><span style="background-color: white; color: black;"><img alt="image" src="https://private-user-images.githubusercontent.com/7347994/302104152-5986f2da-ff44-4a02-9546-1ae866bb39f7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNDE1Mi01OTg2ZjJkYS1mZjQ0LTRhMDItOTU0Ni0xYWU4NjZiYjM5ZjcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YmIzMjVlOGUyN2IxNzM1MTY1OWYxMjJmNzQ3ZTNjODc0YWMzYWVkM2YxYjI3YmUzNmMyNjYzODk0MzBhMzllNyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.R-NICRF-NqmdYowdKXEnsYWaDLFlqsPaJSihWvBJvBg" style="border-style: none; box-sizing: content-box; max-width: 100%;" /></span></a></p><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;"><span style="background-color: white;">Run the setup file (LM-Studio-0.2.12-Setup.exe) to install LM Studio locally.</span></p></li><li style="box-sizing: border-box;"><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;"><span style="background-color: white;">Once installed, LM Studio will open:</span></p></li><li style="box-sizing: border-box; margin-top: 0.25em;"><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;"><a href="https://private-user-images.githubusercontent.com/7347994/302104489-c0f90407-4478-4885-9e01-6b4f06a68367.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNDQ4OS1jMGY5MDQwNy00NDc4LTQ4ODUtOWUwMS02YjRmMDZhNjgzNjcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9M2YxYmE1ZTljYjk0MjVjNGM3YWY0NDUzZTJjZDZkNzc2NThjODE4MmNkZmE0NTdlODMxM2ZjYmFiMWQ1NDIyNyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.bd47vmPoqGuYPFD-8YbA5dm8Fwgude06oo6vOU9aLUA" rel="noopener noreferrer" style="box-sizing: border-box; text-underline-offset: 0.2rem;" target="_blank"><span style="background-color: white; color: black;"><img alt="image" src="https://private-user-images.githubusercontent.com/7347994/302104489-c0f90407-4478-4885-9e01-6b4f06a68367.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNDQ4OS1jMGY5MDQwNy00NDc4LTQ4ODUtOWUwMS02YjRmMDZhNjgzNjcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9M2YxYmE1ZTljYjk0MjVjNGM3YWY0NDUzZTJjZDZkNzc2NThjODE4MmNkZmE0NTdlODMxM2ZjYmFiMWQ1NDIyNyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.bd47vmPoqGuYPFD-8YbA5dm8Fwgude06oo6vOU9aLUA" style="border-style: none; box-sizing: content-box; max-width: 100%;" /></span></a></p><h2 dir="auto" style="border-bottom: 1px solid var(--borderColor-muted, var(--color-border-muted)); box-sizing: border-box; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;" tabindex="-1"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/Running_Local_LLM#download-llms" id="user-content-download-llms" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; position: absolute; text-decoration-line: none; text-underline-offset: 0.2rem;" tabindex="-1"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span style="background-color: white; color: black;"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></span></a><span style="background-color: white;">Download LLMs:</span></h2><ul dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><span style="background-color: white;">Search for the latest Llama-2 model and install the one that will fit in your machine:</span></li></ul><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;"><a href="https://private-user-images.githubusercontent.com/7347994/302104585-01c39544-df6b-4d19-a2b8-4c3a6e0a6269.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNDU4NS0wMWMzOTU0NC1kZjZiLTRkMTktYTJiOC00YzNhNmUwYTYyNjkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MTUxZGRhNmEzNjYzNmQwZDJkY2JhYTgyMzY5M2YxYWNkMjU1NDQ2ZDRmM2Y1NTZmYmU3NWY0OThhNzFmYzQ3YiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.PnKK-WSmf0yUwa9-YsSw5x606aSzE3ksCu3nmzv_95o" rel="noopener noreferrer" style="box-sizing: border-box; text-underline-offset: 0.2rem;" target="_blank"><span style="background-color: white; color: black;"><img alt="image" src="https://private-user-images.githubusercontent.com/7347994/302104585-01c39544-df6b-4d19-a2b8-4c3a6e0a6269.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNDU4NS0wMWMzOTU0NC1kZjZiLTRkMTktYTJiOC00YzNhNmUwYTYyNjkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MTUxZGRhNmEzNjYzNmQwZDJkY2JhYTgyMzY5M2YxYWNkMjU1NDQ2ZDRmM2Y1NTZmYmU3NWY0OThhNzFmYzQ3YiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.PnKK-WSmf0yUwa9-YsSw5x606aSzE3ksCu3nmzv_95o" style="border-style: none; box-sizing: content-box; max-width: 100%;" /></span></a></p><ul dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><span style="background-color: white;">Navigate to the chat section and select the downloaded model and start chatting!:</span></li></ul><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;"><a href="https://private-user-images.githubusercontent.com/7347994/302105842-aa45b7b7-9c33-4562-8bea-294230b33f04.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNTg0Mi1hYTQ1YjdiNy05YzMzLTQ1NjItOGJlYS0yOTQyMzBiMzNmMDQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ODI3MGY2Njk4YzZhOTI1ODAxNjYyYWViOGE1YzYzMTk4YmFlODk5MGEzYmJkZTc5MTg5MmEzMjVkOGYyNDE3YiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.l0mX50J1w3vRqjggPIsbA7QnqD1bJU3PQ9cBvvHZGEc" rel="noopener noreferrer" style="box-sizing: border-box; text-underline-offset: 0.2rem;" target="_blank"><span style="background-color: white; color: black;"><img alt="image" src="https://private-user-images.githubusercontent.com/7347994/302105842-aa45b7b7-9c33-4562-8bea-294230b33f04.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNTg0Mi1hYTQ1YjdiNy05YzMzLTQ1NjItOGJlYS0yOTQyMzBiMzNmMDQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ODI3MGY2Njk4YzZhOTI1ODAxNjYyYWViOGE1YzYzMTk4YmFlODk5MGEzYmJkZTc5MTg5MmEzMjVkOGYyNDE3YiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.l0mX50J1w3vRqjggPIsbA7QnqD1bJU3PQ9cBvvHZGEc" style="border-style: none; box-sizing: content-box; max-width: 100%;" /></span></a></p><h2 dir="auto" style="border-bottom: 1px solid var(--borderColor-muted, var(--color-border-muted)); box-sizing: border-box; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;" tabindex="-1"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/Running_Local_LLM#enable-the-local-inference-server" id="user-content-enable-the-local-inference-server" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; position: absolute; text-decoration-line: none; text-underline-offset: 0.2rem;" tabindex="-1"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span style="background-color: white; color: black;"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></span></a><span style="background-color: white;">Enable the local inference server</span></h2><ul dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><span style="background-color: white;">We can expose this model so we can access it programmatically:</span></li><li style="box-sizing: border-box; margin-top: 0.25em;"><span style="background-color: white;">Navigate to the local server tab and start the server on port 1234.</span></li></ul><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;"><a href="https://private-user-images.githubusercontent.com/7347994/302105998-63c84905-01e4-4a7a-b97c-9dce50f2a564.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNTk5OC02M2M4NDkwNS0wMWU0LTRhN2EtYjk3Yy05ZGNlNTBmMmE1NjQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MDgzYjQ2ZTllNzZhMTMwMWE4Yzk1Y2UwMmY2MzAyM2JjMjAwOTFmZDdmZDI1ZmM1ZTJmOWZlOTc1MGZkYmI2ZSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.5jvXyp5Ju11Vk0aE8PCbql6ZxOMWRfNqpu8VtRwwX90" rel="noopener noreferrer" style="box-sizing: border-box; text-underline-offset: 0.2rem;" target="_blank"><span style="background-color: white; color: black;"><img alt="image" src="https://private-user-images.githubusercontent.com/7347994/302105998-63c84905-01e4-4a7a-b97c-9dce50f2a564.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNTk5OC02M2M4NDkwNS0wMWU0LTRhN2EtYjk3Yy05ZGNlNTBmMmE1NjQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MDgzYjQ2ZTllNzZhMTMwMWE4Yzk1Y2UwMmY2MzAyM2JjMjAwOTFmZDdmZDI1ZmM1ZTJmOWZlOTc1MGZkYmI2ZSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.5jvXyp5Ju11Vk0aE8PCbql6ZxOMWRfNqpu8VtRwwX90" style="border-style: none; box-sizing: content-box; max-width: 100%;" /></span></a></p><h2 dir="auto" style="border-bottom: 1px solid var(--borderColor-muted, var(--color-border-muted)); box-sizing: border-box; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;" tabindex="-1"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/Running_Local_LLM#use-openai-api-to-talk-to-the-model" id="user-content-use-openai-api-to-talk-to-the-model" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; position: absolute; text-decoration-line: none; text-underline-offset: 0.2rem;" tabindex="-1"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span style="background-color: white; color: black;"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></span></a><span style="background-color: white;">Use OpenAI API to talk to the model</span></h2><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">Get my Jupyter Notebook and run it:</span></p><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/Running_Local_LLM/blob/main/Running_LLM_Locally.ipynb" style="box-sizing: border-box; text-underline-offset: 0.2rem;"><span style="background-color: white; color: black;">https://github.com/JordiCorbilla/Running_Local_LLM/blob/main/Running_LLM_Locally.ipynb</span></a></p><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 0px;"><a href="https://private-user-images.githubusercontent.com/7347994/302106424-d5e3c39c-6078-408a-a3c9-826262d585ee.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNjQyNC1kNWUzYzM5Yy02MDc4LTQwOGEtYTNjOS04MjYyNjJkNTg1ZWUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9N2NlMTg4MGU2NjRlMzBkN2M2MzRmOTFhZTI3OGQxMTBlMzgzYmJmN2QyMWZiOGU5YzkzMTVlOTRmOWU5N2FiYSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.oHTm16nxz6_I3LV2j6t77NwXdAQmaOAt8NMUlEPa6Vs" rel="noopener noreferrer" style="box-sizing: border-box; text-underline-offset: 0.2rem;" target="_blank"><span style="background-color: white; color: black;"><img alt="image" src="https://private-user-images.githubusercontent.com/7347994/302106424-d5e3c39c-6078-408a-a3c9-826262d585ee.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4OTAyMjksIm5iZiI6MTcwODg4OTkyOSwicGF0aCI6Ii83MzQ3OTk0LzMwMjEwNjQyNC1kNWUzYzM5Yy02MDc4LTQwOGEtYTNjOS04MjYyNjJkNTg1ZWUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIyNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMjVUMTkzODQ5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9N2NlMTg4MGU2NjRlMzBkN2M2MzRmOTFhZTI3OGQxMTBlMzgzYmJmN2QyMWZiOGU5YzkzMTVlOTRmOWU5N2FiYSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.oHTm16nxz6_I3LV2j6t77NwXdAQmaOAt8NMUlEPa6Vs" style="border-style: none; box-sizing: content-box; max-width: 100%;" /></span></a></p><p dir="auto" style="box-sizing: border-box; margin-bottom: 0px !important; margin-top: 0px;"><span style="background-color: white;">Now you can talk to the LLM locally and you can even expose this externally using <b>ngrok</b>.</span></p></li></ul>Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-1660506083740980672024-02-11T18:25:00.006+01:002024-02-11T19:32:43.035+01:00OpenAI WhatsApp Bot with Python<p><span><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; font-size: 16px;"></span></span></p><div class="separator" style="clear: both; text-align: center;"><span><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; font-size: 16px;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk-FSq2zAVZ_DSQNCcoLGkfsrUU0Y7pOH3SnKPW_iLkq26IC8aWInKBs7Q59B9FTOSBD_20hL1nZKdXRBg7uxpSWhlpKaU139bG74omueDT1W1-hAkHUh0ZsXBbq0n8fOWespsvQdeX6DL9Uaj5fpJ4OgYhxK-f5rYnr7y0QHECxtDSds1ziOkM9HXv8I/s1024/DALL%C2%B7E%202024-02-11%2018.23.17%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_OpenAI%20WhatsApp%20Bot%20with%20Python_.%20The%20image%20should%20visually%20represent%20the%20integration%20of%20OpenAI's%20tec.webp" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1024" data-original-width="1024" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk-FSq2zAVZ_DSQNCcoLGkfsrUU0Y7pOH3SnKPW_iLkq26IC8aWInKBs7Q59B9FTOSBD_20hL1nZKdXRBg7uxpSWhlpKaU139bG74omueDT1W1-hAkHUh0ZsXBbq0n8fOWespsvQdeX6DL9Uaj5fpJ4OgYhxK-f5rYnr7y0QHECxtDSds1ziOkM9HXv8I/w246-h246/DALL%C2%B7E%202024-02-11%2018.23.17%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_OpenAI%20WhatsApp%20Bot%20with%20Python_.%20The%20image%20should%20visually%20represent%20the%20integration%20of%20OpenAI's%20tec.webp" width="246" /></a></span></span></div><span><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; font-size: 16px;"><br />The "OpenAI-Whatsapp-Bot" is a project focused on integrating OpenAI's language models with WhatsApp. This Python-based bot enables users to interact with AI directly through WhatsApp messages. Repo can be found <a href="https://github.com/JordiCorbilla/OpenAI-Whatsapp-Bot">here</a>.</span></span><p></p><p><span><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; font-size: 16px;"><br /></span></span></p><p><br /></p><p><span><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; font-size: 16px;"><br /></span></span></p><p><br /></p><p><br /></p><h1 dir="auto" style="border-bottom: 1px solid var(--borderColor-muted, var(--color-border-muted)); box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; margin: 0px 0px 16px; padding-bottom: 0.3em;" tabindex="-1"><span style="background-color: white; font-weight: var(--base-text-weight-semibold, 600);">Setup environment:</span></h1><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><span style="background-color: white;">Windows 11 PRO (Mini Server x64 Intel Celeron 12 Gb)</span></li><li style="box-sizing: border-box; margin-top: 0.25em;"><span style="background-color: white;">Python 3.12.1</span></li><li style="box-sizing: border-box; margin-top: 0.25em;"><span style="background-color: white;">Sql Server Express v15</span></li><li style="box-sizing: border-box; margin-top: 0.25em;"><span style="background-color: white;">Twilio Account</span></li><li style="box-sizing: border-box; margin-top: 0.25em;"><span style="background-color: white;">Ngrok</span></li><li style="box-sizing: border-box; margin-top: 0.25em;"><span style="background-color: white;">OpenAPI Key</span></li><li style="box-sizing: border-box; margin-top: 0.25em;"><span style="background-color: white;">Whatsapp (local phone)</span></li></ul><h2 dir="auto" style="border-bottom: 1px solid var(--borderColor-muted, var(--color-border-muted)); box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;" tabindex="-1"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/OpenAI-Whatsapp-Bot?tab=readme-ov-file#1-setting-up-your-twilio-account" id="user-content-1-setting-up-your-twilio-account" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; position: absolute; text-decoration-line: none; text-underline-offset: 0.2rem;" tabindex="-1"><span><span style="background-color: white; color: black;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></span></span></span></span></a><span style="background-color: white;">1) Setting up your Twilio Account</span></h2><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><span style="background-color: white;">Create a free account with Twilio here -> <a href="https://www.twilio.com/en-us" rel="nofollow" style="box-sizing: border-box; text-underline-offset: 0.2rem;">https://www.twilio.com/en-us</a></span></li><li style="box-sizing: border-box; margin-top: 0.25em;"><span style="background-color: white;">Setup the WhatsApp messaging using the Twilio Sandbox, you'll be given a <code style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em; white-space-collapse: break-spaces;">+1</code> phone number to interact with:</span></li></ul><div><span style="font-family: -apple-system, BlinkMacSystemFont, Segoe UI, Noto Sans, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEj1l9QTE-_vj302LTNhKAth9YwC612pZStguN2pwQ8JDQreTHa5Lcdzy_mqM14PDhI4cwoWvLKerx1XXXWXdmNWiN7yjssAcK6FNrADj7a7z9PPZiZtEzkc98OEuzv9YgwZnI3Wi2Qz7hTyHRWvh1wjkCgYatVsinUkhUi47udUzjTUDlT9JIq9hLWuuT8" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1133" data-original-width="807" src="https://blogger.googleusercontent.com/img/a/AVvXsEj1l9QTE-_vj302LTNhKAth9YwC612pZStguN2pwQ8JDQreTHa5Lcdzy_mqM14PDhI4cwoWvLKerx1XXXWXdmNWiN7yjssAcK6FNrADj7a7z9PPZiZtEzkc98OEuzv9YgwZnI3Wi2Qz7hTyHRWvh1wjkCgYatVsinUkhUi47udUzjTUDlT9JIq9hLWuuT8=s16000" /></a></div><br /></span></div><h2 dir="auto" style="border-bottom: 1px solid var(--borderColor-muted, var(--color-border-muted)); box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;" tabindex="-1"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/OpenAI-Whatsapp-Bot?tab=readme-ov-file#2-install-python-and-create-your-development-environment" id="user-content-2-install-python-and-create-your-development-environment" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; position: absolute; text-decoration-line: none; text-underline-offset: 0.2rem;" tabindex="-1"><span><span style="background-color: white; color: black;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></span></span></span></span></a><span style="background-color: white;">2) Install Python and create your development environment</span></h2><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">I've done this using the latest python available <span style="box-sizing: border-box; font-weight: var(--base-text-weight-semibold, 600);">3.12.1</span> (<a href="https://www.python.org/ftp/python/3.12.1/python-3.12.1-amd64.exe" rel="nofollow" style="box-sizing: border-box; text-underline-offset: 0.2rem;">https://www.python.org/ftp/python/3.12.1/python-3.12.1-amd64.exe</a>) at the time of this writing.</span></p><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;"><span style="background-color: white;">Clone this repo: <a href="https://github.com/JordiCorbilla/OpenAI-Whatsapp-Bot.git" style="box-sizing: border-box; text-underline-offset: 0.2rem;">https://github.com/JordiCorbilla/OpenAI-Whatsapp-Bot.git</a></span></p></li><li style="box-sizing: border-box; margin-top: 0.25em;"><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;"><span style="background-color: white;">Create the dev environment:</span></p></li></ul><div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" style="box-sizing: border-box; display: flex; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; justify-content: space-between; margin-bottom: 16px; overflow: auto; position: relative;"><pre style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; min-height: 52px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span style="background-color: white;">C:<span class="pl-cce" style="box-sizing: border-box;">\r</span>epo<span class="pl-cce" style="box-sizing: border-box;">\O</span>penAI-Whatsapp-Bot<span class="pl-k" style="box-sizing: border-box;">></span> python -m venv venv<span class="pl-k" style="box-sizing: border-box;">;</span></span></pre><div class="zeroclipboard-container" style="animation: auto ease 0s 1 normal none running none; box-sizing: border-box;"><clipboard-copy aria-label="Copy" class="ClipboardButton btn btn-invisible js-clipboard-copy m-2 p-0 tooltipped-no-delay d-flex flex-justify-center flex-items-center" data-copy-feedback="Copied!" data-tooltip-direction="w" role="button" style="align-items: center; appearance: none; border-radius: 6px; border: 0px; box-shadow: none; box-sizing: border-box; cursor: pointer; display: flex; font-size: 14px; font-weight: var(--base-text-weight-medium, 500); height: var(--control-small-size, 28px); justify-content: center; line-height: 20px; margin: var(--base-size-8, 8px) !important; padding: 0px; position: relative; text-wrap: nowrap; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s; user-select: none; vertical-align: middle; width: var(--control-small-size, 28px);" tabindex="0" value="C:\repo\OpenAI-Whatsapp-Bot> python -m venv venv;"><svg aria-hidden="true" class="octicon octicon-copy js-clipboard-copy-icon" data-view-component="true" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><span style="background-color: white;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></span></span></span></clipboard-copy></div></div><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><span style="background-color: white;">Enable the dev environment:</span></li></ul><div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" style="box-sizing: border-box; display: flex; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; justify-content: space-between; margin-bottom: 16px; overflow: auto; position: relative;"><pre style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; min-height: 52px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span style="background-color: white;">C:<span class="pl-cce" style="box-sizing: border-box;">\r</span>epo<span class="pl-cce" style="box-sizing: border-box;">\O</span>penAI-Whatsapp-Bot<span class="pl-cce" style="box-sizing: border-box;">\v</span>env<span class="pl-cce" style="box-sizing: border-box;">\S</span>cripts<span class="pl-k" style="box-sizing: border-box;">></span>activate.bat</span></pre><div class="zeroclipboard-container" style="animation: auto ease 0s 1 normal none running none; box-sizing: border-box;"><clipboard-copy aria-label="Copy" class="ClipboardButton btn btn-invisible js-clipboard-copy m-2 p-0 tooltipped-no-delay d-flex flex-justify-center flex-items-center" data-copy-feedback="Copied!" data-tooltip-direction="w" role="button" style="align-items: center; appearance: none; border-radius: 6px; border: 0px; box-shadow: none; box-sizing: border-box; cursor: pointer; display: flex; font-size: 14px; font-weight: var(--base-text-weight-medium, 500); height: var(--control-small-size, 28px); justify-content: center; line-height: 20px; margin: var(--base-size-8, 8px) !important; padding: 0px; position: relative; text-wrap: nowrap; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s; user-select: none; vertical-align: middle; width: var(--control-small-size, 28px);" tabindex="0" value="C:\repo\OpenAI-Whatsapp-Bot\venv\Scripts>activate.bat"><svg aria-hidden="true" class="octicon octicon-copy js-clipboard-copy-icon" data-view-component="true" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><span style="background-color: white;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></span></span></span></clipboard-copy></div></div><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><span style="background-color: white;">Upgrade pip</span></li></ul><div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" style="box-sizing: border-box; display: flex; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; justify-content: space-between; margin-bottom: 16px; overflow: auto; position: relative;"><pre style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; min-height: 52px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span style="background-color: white;">(venv) C:<span class="pl-cce" style="box-sizing: border-box;">\r</span>epo<span class="pl-cce" style="box-sizing: border-box;">\O</span>penAI-Whatsapp-Bot<span class="pl-cce" style="box-sizing: border-box;">\v</span>env<span class="pl-cce" style="box-sizing: border-box;">\S</span>cripts<span class="pl-k" style="box-sizing: border-box;">></span>python.exe -m pip install --upgrade pip
Requirement already satisfied: pip <span class="pl-k" style="box-sizing: border-box;">in</span> c:<span class="pl-cce" style="box-sizing: border-box;">\r</span>epo<span class="pl-cce" style="box-sizing: border-box;">\o</span>penai-whatsapp-bot<span class="pl-cce" style="box-sizing: border-box;">\v</span>env<span class="pl-cce" style="box-sizing: border-box;">\l</span>ib<span class="pl-cce" style="box-sizing: border-box;">\s</span>ite-packages (23.2.1)
Collecting pip
Obtaining dependency information <span class="pl-k" style="box-sizing: border-box;">for</span> pip from https://files.pythonhosted.org/packages/15/aa/3f4c7bcee2057a76562a5b33ecbd199be08cdb4443a02e26bd2c3cf6fc39/pip-23.3.2-py3-none-any.whl.metadata
Using cached pip-23.3.2-py3-none-any.whl.metadata (3.5 kB)
Using cached pip-23.3.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 23.2.1
Uninstalling pip-23.2.1:
Successfully uninstalled pip-23.2.1
Successfully installed pip-23.3.2</span></pre><div class="zeroclipboard-container" style="animation: auto ease 0s 1 normal none running none; box-sizing: border-box;"><clipboard-copy aria-label="Copy" class="ClipboardButton btn btn-invisible js-clipboard-copy m-2 p-0 tooltipped-no-delay d-flex flex-justify-center flex-items-center" data-copy-feedback="Copied!" data-tooltip-direction="w" role="button" style="align-items: center; appearance: none; border-radius: 6px; border: 0px; box-shadow: none; box-sizing: border-box; cursor: pointer; display: flex; font-size: 14px; font-weight: var(--base-text-weight-medium, 500); height: var(--control-small-size, 28px); justify-content: center; line-height: 20px; margin: var(--base-size-8, 8px) !important; padding: 0px; position: relative; text-wrap: nowrap; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s; user-select: none; vertical-align: middle; width: var(--control-small-size, 28px);" tabindex="0" value="(venv) C:\repo\OpenAI-Whatsapp-Bot\venv\Scripts>python.exe -m pip install --upgrade pip
Requirement already satisfied: pip in c:\repo\openai-whatsapp-bot\venv\lib\site-packages (23.2.1)
Collecting pip
Obtaining dependency information for pip from https://files.pythonhosted.org/packages/15/aa/3f4c7bcee2057a76562a5b33ecbd199be08cdb4443a02e26bd2c3cf6fc39/pip-23.3.2-py3-none-any.whl.metadata
Using cached pip-23.3.2-py3-none-any.whl.metadata (3.5 kB)
Using cached pip-23.3.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 23.2.1
Uninstalling pip-23.2.1:
Successfully uninstalled pip-23.2.1
Successfully installed pip-23.3.2"><svg aria-hidden="true" class="octicon octicon-copy js-clipboard-copy-icon" data-view-component="true" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><span style="background-color: white;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></span></span></span></clipboard-copy></div></div><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;"><span style="background-color: white;">Install Microsoft C++ Build Tools </span></p></li></ul><div><span style="font-family: -apple-system, BlinkMacSystemFont, Segoe UI, Noto Sans, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiSxhMlFtrbVja_6qol-5qzn1cfR5NbXEeVHF8oWaOg-KeerNRFnaZzAhbKqM6Txt9h1Zwn1sXQr2PSmCclKozwVvhtF62fuO1_u4lxvQEUUW1P-cm4gYaW4tPal2zngUaHYNSDoXbtTW2HxLW8amIm__sNWIc-rWvcTnlu7Lo60P2s5SWTGZRuOR0A4uQ" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="723" data-original-width="1279" height="362" src="https://blogger.googleusercontent.com/img/a/AVvXsEiSxhMlFtrbVja_6qol-5qzn1cfR5NbXEeVHF8oWaOg-KeerNRFnaZzAhbKqM6Txt9h1Zwn1sXQr2PSmCclKozwVvhtF62fuO1_u4lxvQEUUW1P-cm4gYaW4tPal2zngUaHYNSDoXbtTW2HxLW8amIm__sNWIc-rWvcTnlu7Lo60P2s5SWTGZRuOR0A4uQ=w640-h362" width="640" /></a></div><br /><br /></span></div><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box; margin-top: 0.25em;"><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;"><span style="background-color: white;">Install the requirements file using <code style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em; white-space-collapse: break-spaces;">pip install -r requirements.txt</code>:</span></p></li></ul><div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" style="box-sizing: border-box; display: flex; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; justify-content: space-between; margin-bottom: 16px; overflow: auto; position: relative;"><pre style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; min-height: 52px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span style="background-color: white;">aiohttp==3.9.1
aiohttp-retry==2.8.3
aiosignal==1.3.1
annotated-types==0.6.0
anyio==3.7.1
attrs==23.2.0
certifi==2023.11.17
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
distro==1.9.0
fastapi==0.109.0
frozenlist==1.4.1
greenlet==3.0.3
h11==0.14.0
httpcore==1.0.2
httpx==0.26.0
idna==3.6
iniconfig==2.0.0
multidict==6.0.4
openai==0.28.0
packaging==23.2
pluggy==1.3.0
psycopg2-binary==2.9.9
pydantic==2.5.3
pydantic_core==2.14.6
PyJWT==2.8.0
pyngrok==7.0.5
pyodbc==5.0.1
pytest==7.4.4
python-decouple==3.8
python-multipart==0.0.6
PyYAML==6.0.1
requests==2.31.0
sniffio==1.3.0
SQLAlchemy==2.0.25
starlette==0.35.1
tqdm==4.66.1
twilio==8.11.1
typing_extensions==4.9.0
urllib3==2.1.0
uvicorn==0.26.0
yarl==1.9.4</span></pre><div class="zeroclipboard-container" style="animation: auto ease 0s 1 normal none running none; box-sizing: border-box;"><clipboard-copy aria-label="Copy" class="ClipboardButton btn btn-invisible js-clipboard-copy m-2 p-0 tooltipped-no-delay d-flex flex-justify-center flex-items-center" data-copy-feedback="Copied!" data-tooltip-direction="w" role="button" style="align-items: center; appearance: none; border-radius: 6px; border: 0px; box-shadow: none; box-sizing: border-box; cursor: pointer; display: flex; font-size: 14px; font-weight: var(--base-text-weight-medium, 500); height: var(--control-small-size, 28px); justify-content: center; line-height: 20px; margin: var(--base-size-8, 8px) !important; padding: 0px; position: relative; text-wrap: nowrap; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s; user-select: none; vertical-align: middle; width: var(--control-small-size, 28px);" tabindex="0" value="aiohttp==3.9.1
aiohttp-retry==2.8.3
aiosignal==1.3.1
annotated-types==0.6.0
anyio==3.7.1
attrs==23.2.0
certifi==2023.11.17
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
distro==1.9.0
fastapi==0.109.0
frozenlist==1.4.1
greenlet==3.0.3
h11==0.14.0
httpcore==1.0.2
httpx==0.26.0
idna==3.6
iniconfig==2.0.0
multidict==6.0.4
openai==0.28.0
packaging==23.2
pluggy==1.3.0
psycopg2-binary==2.9.9
pydantic==2.5.3
pydantic_core==2.14.6
PyJWT==2.8.0
pyngrok==7.0.5
pyodbc==5.0.1
pytest==7.4.4
python-decouple==3.8
python-multipart==0.0.6
PyYAML==6.0.1
requests==2.31.0
sniffio==1.3.0
SQLAlchemy==2.0.25
starlette==0.35.1
tqdm==4.66.1
twilio==8.11.1
typing_extensions==4.9.0
urllib3==2.1.0
uvicorn==0.26.0
yarl==1.9.4"><svg aria-hidden="true" class="octicon octicon-copy js-clipboard-copy-icon" data-view-component="true" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><span style="background-color: white;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></span></span></span></clipboard-copy></div></div><div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" style="box-sizing: border-box; display: flex; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; justify-content: space-between; margin-bottom: 16px; overflow: auto; position: relative;"><pre style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; min-height: 52px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span style="background-color: white;">(venv) C:<span class="pl-cce" style="box-sizing: border-box;">\r</span>epo<span class="pl-cce" style="box-sizing: border-box;">\O</span>penAI-Whatsapp-Bot<span class="pl-k" style="box-sizing: border-box;">></span>pip install -r requirements.txt
Collecting fastapi (from -r requirements.txt (line 1))
Using cached fastapi-0.109.0-py3-none-any.whl.metadata (24 kB)
Collecting uvicorn (from -r requirements.txt (line 2))
Using cached uvicorn-0.26.0-py3-none-any.whl.metadata (6.4 kB)
Collecting twilio (from -r requirements.txt (line 3))
Using cached twilio-8.11.1-py2.py3-none-any.whl.metadata (12 kB)
Collecting openai (from -r requirements.txt (line 4))
Using cached openai-1.9.0-py3-none-any.whl.metadata (18 kB)
Collecting python-decouple (from -r requirements.txt (line 5))
Using cached python_decouple-3.8-py3-none-any.whl (9.9 kB)
Collecting sqlalchemy (from -r requirements.txt (line 6))
Using cached SQLAlchemy-2.0.25-cp312-cp312-win_amd64.whl.metadata (9.8 kB)
Collecting psycopg2-binary (from -r requirements.txt (line 7))
Using cached psycopg2_binary-2.9.9-cp312-cp312-win_amd64.whl.metadata (4.6 kB)
Collecting python-multipart (from -r requirements.txt (line 8))
Using cached python_multipart-0.0.6-py3-none-any.whl (45 kB)
Collecting pyngrok (from -r requirements.txt (line 9))
Using cached pyngrok-7.0.5-py3-none-any.whl.metadata (6.2 kB)
Collecting pydantic<span class="pl-k" style="box-sizing: border-box;">!</span>=1.8,<span class="pl-k" style="box-sizing: border-box;">!</span>=1.8.1,<span class="pl-k" style="box-sizing: border-box;">!</span>=2.0.0,<span class="pl-k" style="box-sizing: border-box;">!</span>=2.0.1,<span class="pl-k" style="box-sizing: border-box;">!</span>=2.1.0,<span class="pl-k" style="box-sizing: border-box;"><</span>3.0.0,<span class="pl-k" style="box-sizing: border-box;">></span>=1.7.4 (from fastapi-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 1))
Using cached pydantic-2.5.3-py3-none-any.whl.metadata (65 kB)
Collecting starlette<span class="pl-k" style="box-sizing: border-box;"><</span>0.36.0,<span class="pl-k" style="box-sizing: border-box;">></span>=0.35.0 (from fastapi-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 1))
Using cached starlette-0.35.1-py3-none-any.whl.metadata (5.8 kB)
Collecting typing-extensions<span class="pl-k" style="box-sizing: border-box;">></span>=4.8.0 (from fastapi-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 1))
Using cached typing_extensions-4.9.0-py3-none-any.whl.metadata (3.0 kB)
Collecting click<span class="pl-k" style="box-sizing: border-box;">></span>=7.0 (from uvicorn-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 2))
Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting h<span class="pl-k" style="box-sizing: border-box;">11></span>=0.8 (from uvicorn-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 2))
Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting requests<span class="pl-k" style="box-sizing: border-box;">></span>=2.0.0 (from twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting PyJWT<span class="pl-k" style="box-sizing: border-box;"><</span>3.0.0,<span class="pl-k" style="box-sizing: border-box;">></span>=2.0.0 (from twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached PyJWT-2.8.0-py3-none-any.whl.metadata (4.2 kB)
Collecting aiohttp<span class="pl-k" style="box-sizing: border-box;">></span>=3.8.4 (from twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached aiohttp-3.9.1-cp312-cp312-win_amd64.whl.metadata (7.6 kB)
Collecting aiohttp-retry<span class="pl-k" style="box-sizing: border-box;">></span>=2.8.3 (from twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached aiohttp_retry-2.8.3-py3-none-any.whl (9.8 kB)
Collecting anyio<span class="pl-k" style="box-sizing: border-box;"><</span>5,<span class="pl-k" style="box-sizing: border-box;">></span>=3.5.0 (from openai-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 4))
Using cached anyio-4.2.0-py3-none-any.whl.metadata (4.6 kB)
Collecting distro<span class="pl-k" style="box-sizing: border-box;"><</span>2,<span class="pl-k" style="box-sizing: border-box;">></span>=1.7.0 (from openai-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 4))
Using cached distro-1.9.0-py3-none-any.whl.metadata (6.8 kB)
Collecting httpx<span class="pl-k" style="box-sizing: border-box;"><</span>1,<span class="pl-k" style="box-sizing: border-box;">></span>=0.23.0 (from openai-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 4))
Using cached httpx-0.26.0-py3-none-any.whl.metadata (7.6 kB)
Collecting sniffio (from openai-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 4))
Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting tqdm<span class="pl-k" style="box-sizing: border-box;">></span>4 (from openai-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 4))
Using cached tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)
Collecting greenlet<span class="pl-k" style="box-sizing: border-box;">!</span>=0.4.17 (from sqlalchemy-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 6))
Using cached greenlet-3.0.3-cp312-cp312-win_amd64.whl.metadata (3.9 kB)
Collecting PyYAML (from pyngrok-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 9))
Using cached PyYAML-6.0.1-cp312-cp312-win_amd64.whl.metadata (2.1 kB)
Collecting attrs<span class="pl-k" style="box-sizing: border-box;">></span>=17.3.0 (from aiohttp<span class="pl-k" style="box-sizing: border-box;">></span>=3.8.4-<span class="pl-k" style="box-sizing: border-box;">></span>twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting multidict<span class="pl-k" style="box-sizing: border-box;"><</span>7.0,<span class="pl-k" style="box-sizing: border-box;">></span>=4.5 (from aiohttp<span class="pl-k" style="box-sizing: border-box;">></span>=3.8.4-<span class="pl-k" style="box-sizing: border-box;">></span>twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached multidict-6.0.4.tar.gz (51 kB)
Installing build dependencies ... <span class="pl-k" style="box-sizing: border-box;">done</span>
Getting requirements to build wheel ... <span class="pl-k" style="box-sizing: border-box;">done</span>
Installing backend dependencies ... <span class="pl-k" style="box-sizing: border-box;">done</span>
Preparing metadata (pyproject.toml) ... <span class="pl-k" style="box-sizing: border-box;">done</span>
Collecting yarl<span class="pl-k" style="box-sizing: border-box;"><</span>2.0,<span class="pl-k" style="box-sizing: border-box;">></span>=1.0 (from aiohttp<span class="pl-k" style="box-sizing: border-box;">></span>=3.8.4-<span class="pl-k" style="box-sizing: border-box;">></span>twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached yarl-1.9.4-cp312-cp312-win_amd64.whl.metadata (32 kB)
Collecting frozenlist<span class="pl-k" style="box-sizing: border-box;">></span>=1.1.1 (from aiohttp<span class="pl-k" style="box-sizing: border-box;">></span>=3.8.4-<span class="pl-k" style="box-sizing: border-box;">></span>twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached frozenlist-1.4.1-cp312-cp312-win_amd64.whl.metadata (12 kB)
Collecting aiosignal<span class="pl-k" style="box-sizing: border-box;">></span>=1.1.2 (from aiohttp<span class="pl-k" style="box-sizing: border-box;">></span>=3.8.4-<span class="pl-k" style="box-sizing: border-box;">></span>twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting idna<span class="pl-k" style="box-sizing: border-box;">></span>=2.8 (from anyio<span class="pl-k" style="box-sizing: border-box;"><</span>5,<span class="pl-k" style="box-sizing: border-box;">></span>=3.5.0-<span class="pl-k" style="box-sizing: border-box;">></span>openai-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 4))
Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting colorama (from click<span class="pl-k" style="box-sizing: border-box;">></span>=7.0-<span class="pl-k" style="box-sizing: border-box;">></span>uvicorn-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 2))
Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting certifi (from httpx<span class="pl-k" style="box-sizing: border-box;"><</span>1,<span class="pl-k" style="box-sizing: border-box;">></span>=0.23.0-<span class="pl-k" style="box-sizing: border-box;">></span>openai-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 4))
Using cached certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB)
Collecting httpcore==1.<span class="pl-k" style="box-sizing: border-box;">*</span> (from httpx<span class="pl-k" style="box-sizing: border-box;"><</span>1,<span class="pl-k" style="box-sizing: border-box;">></span>=0.23.0-<span class="pl-k" style="box-sizing: border-box;">></span>openai-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 4))
Using cached httpcore-1.0.2-py3-none-any.whl.metadata (20 kB)
Collecting annotated-types<span class="pl-k" style="box-sizing: border-box;">></span>=0.4.0 (from pydantic<span class="pl-k" style="box-sizing: border-box;">!</span>=1.8,<span class="pl-k" style="box-sizing: border-box;">!</span>=1.8.1,<span class="pl-k" style="box-sizing: border-box;">!</span>=2.0.0,<span class="pl-k" style="box-sizing: border-box;">!</span>=2.0.1,<span class="pl-k" style="box-sizing: border-box;">!</span>=2.1.0,<span class="pl-k" style="box-sizing: border-box;"><</span>3.0.0,<span class="pl-k" style="box-sizing: border-box;">></span>=1.7.4-<span class="pl-k" style="box-sizing: border-box;">></span>fastapi-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 1))
Using cached annotated_types-0.6.0-py3-none-any.whl.metadata (12 kB)
Collecting pydantic-core==2.14.6 (from pydantic<span class="pl-k" style="box-sizing: border-box;">!</span>=1.8,<span class="pl-k" style="box-sizing: border-box;">!</span>=1.8.1,<span class="pl-k" style="box-sizing: border-box;">!</span>=2.0.0,<span class="pl-k" style="box-sizing: border-box;">!</span>=2.0.1,<span class="pl-k" style="box-sizing: border-box;">!</span>=2.1.0,<span class="pl-k" style="box-sizing: border-box;"><</span>3.0.0,<span class="pl-k" style="box-sizing: border-box;">></span>=1.7.4-<span class="pl-k" style="box-sizing: border-box;">></span>fastapi-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 1))
Using cached pydantic_core-2.14.6-cp312-none-win_amd64.whl.metadata (6.6 kB)
Collecting charset-normalizer<span class="pl-k" style="box-sizing: border-box;"><</span>4,<span class="pl-k" style="box-sizing: border-box;">></span>=2 (from requests<span class="pl-k" style="box-sizing: border-box;">></span>=2.0.0-<span class="pl-k" style="box-sizing: border-box;">></span>twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached charset_normalizer-3.3.2-cp312-cp312-win_amd64.whl.metadata (34 kB)
Collecting urllib<span class="pl-k" style="box-sizing: border-box;">3<</span>3,<span class="pl-k" style="box-sizing: border-box;">></span>=1.21.1 (from requests<span class="pl-k" style="box-sizing: border-box;">></span>=2.0.0-<span class="pl-k" style="box-sizing: border-box;">></span>twilio-<span class="pl-k" style="box-sizing: border-box;">></span>-r requirements.txt (line 3))
Using cached urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB)
Using cached fastapi-0.109.0-py3-none-any.whl (92 kB)
Using cached uvicorn-0.26.0-py3-none-any.whl (60 kB)
Using cached twilio-8.11.1-py2.py3-none-any.whl (1.7 MB)
Using cached openai-1.9.0-py3-none-any.whl (223 kB)
Using cached SQLAlchemy-2.0.25-cp312-cp312-win_amd64.whl (2.1 MB)
Using cached psycopg2_binary-2.9.9-cp312-cp312-win_amd64.whl (1.2 MB)
Using cached pyngrok-7.0.5-py3-none-any.whl (21 kB)
Using cached aiohttp-3.9.1-cp312-cp312-win_amd64.whl (362 kB)
Using cached anyio-4.2.0-py3-none-any.whl (85 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached distro-1.9.0-py3-none-any.whl (20 kB)
Using cached greenlet-3.0.3-cp312-cp312-win_amd64.whl (293 kB)
Using cached httpx-0.26.0-py3-none-any.whl (75 kB)
Using cached httpcore-1.0.2-py3-none-any.whl (76 kB)
Using cached pydantic-2.5.3-py3-none-any.whl (381 kB)
Using cached pydantic_core-2.14.6-cp312-none-win_amd64.whl (1.9 MB)
Using cached PyJWT-2.8.0-py3-none-any.whl (22 kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Using cached starlette-0.35.1-py3-none-any.whl (71 kB)
Using cached tqdm-4.66.1-py3-none-any.whl (78 kB)
Using cached typing_extensions-4.9.0-py3-none-any.whl (32 kB)
Using cached PyYAML-6.0.1-cp312-cp312-win_amd64.whl (138 kB)
Using cached annotated_types-0.6.0-py3-none-any.whl (12 kB)
Using cached attrs-23.2.0-py3-none-any.whl (60 kB)
Using cached certifi-2023.11.17-py3-none-any.whl (162 kB)
Using cached charset_normalizer-3.3.2-cp312-cp312-win_amd64.whl (100 kB)
Using cached frozenlist-1.4.1-cp312-cp312-win_amd64.whl (50 kB)
Using cached idna-3.6-py3-none-any.whl (61 kB)
Using cached urllib3-2.1.0-py3-none-any.whl (104 kB)
Using cached yarl-1.9.4-cp312-cp312-win_amd64.whl (76 kB)
Building wheels <span class="pl-k" style="box-sizing: border-box;">for</span> collected packages: multidict
Building wheel <span class="pl-k" style="box-sizing: border-box;">for</span> multidict (pyproject.toml) ... <span class="pl-k" style="box-sizing: border-box;">done</span>
Created wheel <span class="pl-k" style="box-sizing: border-box;">for</span> multidict: filename=multidict-6.0.4-cp312-cp312-win_amd64.whl size=28182 sha256=ec4f815cc6a33c1f931ddf3bdfd1ae03acb6868b33a27ed22107bfeb83367e7e
Stored <span class="pl-k" style="box-sizing: border-box;">in</span> directory: c:<span class="pl-cce" style="box-sizing: border-box;">\u</span>sers<span class="pl-cce" style="box-sizing: border-box;">\j</span>ordi<span class="pl-cce" style="box-sizing: border-box;">\a</span>ppdata<span class="pl-cce" style="box-sizing: border-box;">\l</span>ocal<span class="pl-cce" style="box-sizing: border-box;">\p</span>ip<span class="pl-cce" style="box-sizing: border-box;">\c</span>ache<span class="pl-cce" style="box-sizing: border-box;">\w</span>heels<span class="pl-cce" style="box-sizing: border-box;">\f</span>6<span class="pl-cce" style="box-sizing: border-box;">\d</span>8<span class="pl-cce" style="box-sizing: border-box;">\f</span>f<span class="pl-cce" style="box-sizing: border-box;">\3</span>c14a64b8f2ab1aa94ba2888f5a988be6ab446ec5c8d1a82da
Successfully built multidict
Installing collected packages: python-decouple, urllib3, typing-extensions, sniffio, PyYAML, python-multipart, PyJWT, psycopg2-binary, multidict, idna, h11, greenlet, frozenlist, distro, colorama, charset-normalizer, certifi, attrs, annotated-types, yarl, tqdm, sqlalchemy, requests, pyngrok, pydantic-core, httpcore, click, anyio, aiosignal, uvicorn, starlette, pydantic, httpx, aiohttp, openai, fastapi, aiohttp-retry, twilio
Successfully installed PyJWT-2.8.0 PyYAML-6.0.1 aiohttp-3.9.1 aiohttp-retry-2.8.3 aiosignal-1.3.1 annotated-types-0.6.0 anyio-4.2.0 attrs-23.2.0 certifi-2023.11.17 charset-normalizer-3.3.2 click-8.1.7 colorama-0.4.6 distro-1.9.0 fastapi-0.109.0 frozenlist-1.4.1 greenlet-3.0.3 h11-0.14.0 httpcore-1.0.2 httpx-0.26.0 idna-3.6 multidict-6.0.4 openai-1.9.0 psycopg2-binary-2.9.9 pydantic-2.5.3 pydantic-core-2.14.6 pyngrok-7.0.5 python-decouple-3.8 python-multipart-0.0.6 requests-2.31.0 sniffio-1.3.0 sqlalchemy-2.0.25 starlette-0.35.1 tqdm-4.66.1 twilio-8.11.1 typing-extensions-4.9.0 urllib3-2.1.0 uvicorn-0.26.0 yarl-1.9.4</span></pre><div class="zeroclipboard-container" style="animation: auto ease 0s 1 normal none running none; box-sizing: border-box;"><clipboard-copy aria-label="Copy" class="ClipboardButton btn btn-invisible js-clipboard-copy m-2 p-0 tooltipped-no-delay d-flex flex-justify-center flex-items-center" data-copy-feedback="Copied!" data-tooltip-direction="w" role="button" style="align-items: center; appearance: none; border-radius: 6px; border: 0px; box-shadow: none; box-sizing: border-box; cursor: pointer; display: flex; font-size: 14px; font-weight: var(--base-text-weight-medium, 500); height: var(--control-small-size, 28px); justify-content: center; line-height: 20px; margin: var(--base-size-8, 8px) !important; padding: 0px; position: relative; text-wrap: nowrap; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s; user-select: none; vertical-align: middle; width: var(--control-small-size, 28px);" tabindex="0" value="(venv) C:\repo\OpenAI-Whatsapp-Bot>pip install -r requirements.txt
Collecting fastapi (from -r requirements.txt (line 1))
Using cached fastapi-0.109.0-py3-none-any.whl.metadata (24 kB)
Collecting uvicorn (from -r requirements.txt (line 2))
Using cached uvicorn-0.26.0-py3-none-any.whl.metadata (6.4 kB)
Collecting twilio (from -r requirements.txt (line 3))
Using cached twilio-8.11.1-py2.py3-none-any.whl.metadata (12 kB)
Collecting openai (from -r requirements.txt (line 4))
Using cached openai-1.9.0-py3-none-any.whl.metadata (18 kB)
Collecting python-decouple (from -r requirements.txt (line 5))
Using cached python_decouple-3.8-py3-none-any.whl (9.9 kB)
Collecting sqlalchemy (from -r requirements.txt (line 6))
Using cached SQLAlchemy-2.0.25-cp312-cp312-win_amd64.whl.metadata (9.8 kB)
Collecting psycopg2-binary (from -r requirements.txt (line 7))
Using cached psycopg2_binary-2.9.9-cp312-cp312-win_amd64.whl.metadata (4.6 kB)
Collecting python-multipart (from -r requirements.txt (line 8))
Using cached python_multipart-0.0.6-py3-none-any.whl (45 kB)
Collecting pyngrok (from -r requirements.txt (line 9))
Using cached pyngrok-7.0.5-py3-none-any.whl.metadata (6.2 kB)
Collecting pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 (from fastapi->-r requirements.txt (line 1))
Using cached pydantic-2.5.3-py3-none-any.whl.metadata (65 kB)
Collecting starlette<0.36.0,>=0.35.0 (from fastapi->-r requirements.txt (line 1))
Using cached starlette-0.35.1-py3-none-any.whl.metadata (5.8 kB)
Collecting typing-extensions>=4.8.0 (from fastapi->-r requirements.txt (line 1))
Using cached typing_extensions-4.9.0-py3-none-any.whl.metadata (3.0 kB)
Collecting click>=7.0 (from uvicorn->-r requirements.txt (line 2))
Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting h11>=0.8 (from uvicorn->-r requirements.txt (line 2))
Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting requests>=2.0.0 (from twilio->-r requirements.txt (line 3))
Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting PyJWT<3.0.0,>=2.0.0 (from twilio->-r requirements.txt (line 3))
Using cached PyJWT-2.8.0-py3-none-any.whl.metadata (4.2 kB)
Collecting aiohttp>=3.8.4 (from twilio->-r requirements.txt (line 3))
Using cached aiohttp-3.9.1-cp312-cp312-win_amd64.whl.metadata (7.6 kB)
Collecting aiohttp-retry>=2.8.3 (from twilio->-r requirements.txt (line 3))
Using cached aiohttp_retry-2.8.3-py3-none-any.whl (9.8 kB)
Collecting anyio<5,>=3.5.0 (from openai->-r requirements.txt (line 4))
Using cached anyio-4.2.0-py3-none-any.whl.metadata (4.6 kB)
Collecting distro<2,>=1.7.0 (from openai->-r requirements.txt (line 4))
Using cached distro-1.9.0-py3-none-any.whl.metadata (6.8 kB)
Collecting httpx<1,>=0.23.0 (from openai->-r requirements.txt (line 4))
Using cached httpx-0.26.0-py3-none-any.whl.metadata (7.6 kB)
Collecting sniffio (from openai->-r requirements.txt (line 4))
Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting tqdm>4 (from openai->-r requirements.txt (line 4))
Using cached tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)
Collecting greenlet!=0.4.17 (from sqlalchemy->-r requirements.txt (line 6))
Using cached greenlet-3.0.3-cp312-cp312-win_amd64.whl.metadata (3.9 kB)
Collecting PyYAML (from pyngrok->-r requirements.txt (line 9))
Using cached PyYAML-6.0.1-cp312-cp312-win_amd64.whl.metadata (2.1 kB)
Collecting attrs>=17.3.0 (from aiohttp>=3.8.4->twilio->-r requirements.txt (line 3))
Using cached attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting multidict<7.0,>=4.5 (from aiohttp>=3.8.4->twilio->-r requirements.txt (line 3))
Using cached multidict-6.0.4.tar.gz (51 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Collecting yarl<2.0,>=1.0 (from aiohttp>=3.8.4->twilio->-r requirements.txt (line 3))
Using cached yarl-1.9.4-cp312-cp312-win_amd64.whl.metadata (32 kB)
Collecting frozenlist>=1.1.1 (from aiohttp>=3.8.4->twilio->-r requirements.txt (line 3))
Using cached frozenlist-1.4.1-cp312-cp312-win_amd64.whl.metadata (12 kB)
Collecting aiosignal>=1.1.2 (from aiohttp>=3.8.4->twilio->-r requirements.txt (line 3))
Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting idna>=2.8 (from anyio<5,>=3.5.0->openai->-r requirements.txt (line 4))
Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting colorama (from click>=7.0->uvicorn->-r requirements.txt (line 2))
Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting certifi (from httpx<1,>=0.23.0->openai->-r requirements.txt (line 4))
Using cached certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB)
Collecting httpcore==1.* (from httpx<1,>=0.23.0->openai->-r requirements.txt (line 4))
Using cached httpcore-1.0.2-py3-none-any.whl.metadata (20 kB)
Collecting annotated-types>=0.4.0 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi->-r requirements.txt (line 1))
Using cached annotated_types-0.6.0-py3-none-any.whl.metadata (12 kB)
Collecting pydantic-core==2.14.6 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi->-r requirements.txt (line 1))
Using cached pydantic_core-2.14.6-cp312-none-win_amd64.whl.metadata (6.6 kB)
Collecting charset-normalizer<4,>=2 (from requests>=2.0.0->twilio->-r requirements.txt (line 3))
Using cached charset_normalizer-3.3.2-cp312-cp312-win_amd64.whl.metadata (34 kB)
Collecting urllib3<3,>=1.21.1 (from requests>=2.0.0->twilio->-r requirements.txt (line 3))
Using cached urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB)
Using cached fastapi-0.109.0-py3-none-any.whl (92 kB)
Using cached uvicorn-0.26.0-py3-none-any.whl (60 kB)
Using cached twilio-8.11.1-py2.py3-none-any.whl (1.7 MB)
Using cached openai-1.9.0-py3-none-any.whl (223 kB)
Using cached SQLAlchemy-2.0.25-cp312-cp312-win_amd64.whl (2.1 MB)
Using cached psycopg2_binary-2.9.9-cp312-cp312-win_amd64.whl (1.2 MB)
Using cached pyngrok-7.0.5-py3-none-any.whl (21 kB)
Using cached aiohttp-3.9.1-cp312-cp312-win_amd64.whl (362 kB)
Using cached anyio-4.2.0-py3-none-any.whl (85 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached distro-1.9.0-py3-none-any.whl (20 kB)
Using cached greenlet-3.0.3-cp312-cp312-win_amd64.whl (293 kB)
Using cached httpx-0.26.0-py3-none-any.whl (75 kB)
Using cached httpcore-1.0.2-py3-none-any.whl (76 kB)
Using cached pydantic-2.5.3-py3-none-any.whl (381 kB)
Using cached pydantic_core-2.14.6-cp312-none-win_amd64.whl (1.9 MB)
Using cached PyJWT-2.8.0-py3-none-any.whl (22 kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Using cached starlette-0.35.1-py3-none-any.whl (71 kB)
Using cached tqdm-4.66.1-py3-none-any.whl (78 kB)
Using cached typing_extensions-4.9.0-py3-none-any.whl (32 kB)
Using cached PyYAML-6.0.1-cp312-cp312-win_amd64.whl (138 kB)
Using cached annotated_types-0.6.0-py3-none-any.whl (12 kB)
Using cached attrs-23.2.0-py3-none-any.whl (60 kB)
Using cached certifi-2023.11.17-py3-none-any.whl (162 kB)
Using cached charset_normalizer-3.3.2-cp312-cp312-win_amd64.whl (100 kB)
Using cached frozenlist-1.4.1-cp312-cp312-win_amd64.whl (50 kB)
Using cached idna-3.6-py3-none-any.whl (61 kB)
Using cached urllib3-2.1.0-py3-none-any.whl (104 kB)
Using cached yarl-1.9.4-cp312-cp312-win_amd64.whl (76 kB)
Building wheels for collected packages: multidict
Building wheel for multidict (pyproject.toml) ... done
Created wheel for multidict: filename=multidict-6.0.4-cp312-cp312-win_amd64.whl size=28182 sha256=ec4f815cc6a33c1f931ddf3bdfd1ae03acb6868b33a27ed22107bfeb83367e7e
Stored in directory: c:\users\jordi\appdata\local\pip\cache\wheels\f6\d8\ff\3c14a64b8f2ab1aa94ba2888f5a988be6ab446ec5c8d1a82da
Successfully built multidict
Installing collected packages: python-decouple, urllib3, typing-extensions, sniffio, PyYAML, python-multipart, PyJWT, psycopg2-binary, multidict, idna, h11, greenlet, frozenlist, distro, colorama, charset-normalizer, certifi, attrs, annotated-types, yarl, tqdm, sqlalchemy, requests, pyngrok, pydantic-core, httpcore, click, anyio, aiosignal, uvicorn, starlette, pydantic, httpx, aiohttp, openai, fastapi, aiohttp-retry, twilio
Successfully installed PyJWT-2.8.0 PyYAML-6.0.1 aiohttp-3.9.1 aiohttp-retry-2.8.3 aiosignal-1.3.1 annotated-types-0.6.0 anyio-4.2.0 attrs-23.2.0 certifi-2023.11.17 charset-normalizer-3.3.2 click-8.1.7 colorama-0.4.6 distro-1.9.0 fastapi-0.109.0 frozenlist-1.4.1 greenlet-3.0.3 h11-0.14.0 httpcore-1.0.2 httpx-0.26.0 idna-3.6 multidict-6.0.4 openai-1.9.0 psycopg2-binary-2.9.9 pydantic-2.5.3 pydantic-core-2.14.6 pyngrok-7.0.5 python-decouple-3.8 python-multipart-0.0.6 requests-2.31.0 sniffio-1.3.0 sqlalchemy-2.0.25 starlette-0.35.1 tqdm-4.66.1 twilio-8.11.1 typing-extensions-4.9.0 urllib3-2.1.0 uvicorn-0.26.0 yarl-1.9.4"><svg aria-hidden="true" class="octicon octicon-copy js-clipboard-copy-icon" data-view-component="true" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><span style="background-color: white;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></span></span></span></clipboard-copy></div></div><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;"><span style="background-color: white;">Install SQL Server Express (<a href="https://www.microsoft.com/en-GB/sql-server/sql-server-downloads" rel="nofollow" style="box-sizing: border-box; text-underline-offset: 0.2rem;">https://www.microsoft.com/en-GB/sql-server/sql-server-downloads</a>) and SQL Server Management Studio (<a href="https://learn.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms?view=sql-server-ver16" rel="nofollow" style="box-sizing: border-box; text-underline-offset: 0.2rem;">https://learn.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms?view=sql-server-ver16</a>):</span></p></li><li style="box-sizing: border-box; margin-top: 0.25em;"><p dir="auto" style="box-sizing: border-box; margin-bottom: 16px; margin-top: 16px;"><span style="background-color: white;">Create a sample DB</span></p></li></ul><div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" style="box-sizing: border-box; display: flex; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; justify-content: space-between; margin-bottom: 16px; overflow: auto; position: relative;"><pre style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; min-height: 52px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span style="background-color: white;"><span class="pl-k" style="box-sizing: border-box;">CREATE</span> <span class="pl-k" style="box-sizing: border-box;">TABLE</span> <span class="pl-en" style="box-sizing: border-box;">Conversations</span>(
id <span class="pl-k" style="box-sizing: border-box;">int</span> identity(<span class="pl-c1" style="box-sizing: border-box;">1</span>,<span class="pl-c1" style="box-sizing: border-box;">1</span>) <span class="pl-k" style="box-sizing: border-box;">primary key</span>,
sender <span class="pl-k" style="box-sizing: border-box;">varchar</span>(max) <span class="pl-k" style="box-sizing: border-box;">null</span>,
message <span class="pl-k" style="box-sizing: border-box;">varchar</span>(max) <span class="pl-k" style="box-sizing: border-box;">null</span>,
response <span class="pl-k" style="box-sizing: border-box;">varchar</span>(max) <span class="pl-k" style="box-sizing: border-box;">null</span>
)</span></pre><div class="zeroclipboard-container" style="animation: auto ease 0s 1 normal none running none; box-sizing: border-box;"><clipboard-copy aria-label="Copy" class="ClipboardButton btn btn-invisible js-clipboard-copy m-2 p-0 tooltipped-no-delay d-flex flex-justify-center flex-items-center" data-copy-feedback="Copied!" data-tooltip-direction="w" role="button" style="align-items: center; appearance: none; border-radius: 6px; border: 0px; box-shadow: none; box-sizing: border-box; cursor: pointer; display: flex; font-size: 14px; font-weight: var(--base-text-weight-medium, 500); height: var(--control-small-size, 28px); justify-content: center; line-height: 20px; margin: var(--base-size-8, 8px) !important; padding: 0px; position: relative; text-wrap: nowrap; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s; user-select: none; vertical-align: middle; width: var(--control-small-size, 28px);" tabindex="0" value="CREATE TABLE Conversations(
id int identity(1,1) primary key,
sender varchar(max) null,
message varchar(max) null,
response varchar(max) null
)"><svg aria-hidden="true" class="octicon octicon-copy js-clipboard-copy-icon" data-view-component="true" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><span style="background-color: white;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></span></span></span></clipboard-copy></div></div><ul dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;"><span style="background-color: white;">Setting up FAST API</span></li></ul><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">Set a barebone API (bot.py):</span></p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" style="box-sizing: border-box; display: flex; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; justify-content: space-between; margin-bottom: 16px; overflow: auto; position: relative;"><pre style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; min-height: 52px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span style="background-color: white;"><span class="pl-k" style="box-sizing: border-box;">from</span> <span class="pl-s1" style="box-sizing: border-box;">fastapi</span> <span class="pl-k" style="box-sizing: border-box;">import</span> <span class="pl-v" style="box-sizing: border-box;">FastAPI</span>
<span class="pl-s1" style="box-sizing: border-box;">app</span> <span class="pl-c1" style="box-sizing: border-box;">=</span> <span class="pl-v" style="box-sizing: border-box;">FastAPI</span>()
<span class="pl-en" style="box-sizing: border-box;">@<span class="pl-s1" style="box-sizing: border-box;">app</span>.<span class="pl-en" style="box-sizing: border-box;">get</span>(<span class="pl-s" style="box-sizing: border-box;">"/"</span>)</span>
<span class="pl-k" style="box-sizing: border-box;">async</span> <span class="pl-k" style="box-sizing: border-box;">def</span> <span class="pl-en" style="box-sizing: border-box;">index</span>():
<span class="pl-k" style="box-sizing: border-box;">return</span> {<span class="pl-s" style="box-sizing: border-box;">"msg"</span>: <span class="pl-s" style="box-sizing: border-box;">"running"</span>}</span></pre><div class="zeroclipboard-container" style="animation: auto ease 0s 1 normal none running none; box-sizing: border-box;"><clipboard-copy aria-label="Copy" class="ClipboardButton btn btn-invisible js-clipboard-copy m-2 p-0 tooltipped-no-delay d-flex flex-justify-center flex-items-center" data-copy-feedback="Copied!" data-tooltip-direction="w" role="button" style="align-items: center; appearance: none; border-radius: 6px; border: 0px; box-shadow: none; box-sizing: border-box; cursor: pointer; display: flex; font-size: 14px; font-weight: var(--base-text-weight-medium, 500); height: var(--control-small-size, 28px); justify-content: center; line-height: 20px; margin: var(--base-size-8, 8px) !important; padding: 0px; position: relative; text-wrap: nowrap; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s; user-select: none; vertical-align: middle; width: var(--control-small-size, 28px);" tabindex="0" value="from fastapi import FastAPI
app = FastAPI()
@app.get("/")
async def index():
return {"msg": "running"}"><svg aria-hidden="true" class="octicon octicon-copy js-clipboard-copy-icon" data-view-component="true" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><span style="background-color: white;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></span></span></span></clipboard-copy></div></div><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">Run the API using <code style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em; white-space-collapse: break-spaces;">uvicorn bot:app --reload</code> in you virtual environment:</span></p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" style="box-sizing: border-box; display: flex; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; justify-content: space-between; margin-bottom: 16px; overflow: auto; position: relative;"><pre class="notranslate" style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px;"><code style="background-attachment: initial; background-clip: initial; background-color: black; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; background: black; border-radius: 6px; border: 0px; box-sizing: border-box; display: inline; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow-wrap: normal; overflow: visible; padding: 0px; word-break: normal;"><span style="background-color: white;">C:\repo\OpenAI-Whatsapp-Bot\venv\Scripts>activate.bat
then:
(venv) C:\repo\OpenAI-Whatsapp-Bot>uvicorn bot:app --reload
INFO: Will watch for changes in these directories: ['C:\\repo\\OpenAI-Whatsapp-Bot']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [248] using StatReload
INFO: Started server process [7056]
INFO: Waiting for application startup.
INFO: Application startup complete.
</span></code></pre><div class="zeroclipboard-container" style="animation: auto ease 0s 1 normal none running none; box-sizing: border-box;"><clipboard-copy aria-label="Copy" class="ClipboardButton btn btn-invisible js-clipboard-copy m-2 p-0 tooltipped-no-delay d-flex flex-justify-center flex-items-center" data-copy-feedback="Copied!" data-tooltip-direction="w" role="button" style="align-items: center; appearance: none; border-radius: 6px; border: 0px; box-shadow: none; box-sizing: border-box; cursor: pointer; display: flex; font-size: 14px; font-weight: var(--base-text-weight-medium, 500); height: var(--control-small-size, 28px); justify-content: center; line-height: 20px; margin: var(--base-size-8, 8px) !important; padding: 0px; position: relative; text-wrap: nowrap; transition: color 80ms cubic-bezier(0.33, 1, 0.68, 1) 0s, background-color 0s ease 0s, box-shadow 0s ease 0s, border-color 0s ease 0s; user-select: none; vertical-align: middle; width: var(--control-small-size, 28px);" tabindex="0" value="C:\repo\OpenAI-Whatsapp-Bot\venv\Scripts>activate.bat
then:
(venv) C:\repo\OpenAI-Whatsapp-Bot>uvicorn bot:app --reload
INFO: Will watch for changes in these directories: ['C:\\repo\\OpenAI-Whatsapp-Bot']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [248] using StatReload
INFO: Started server process [7056]
INFO: Waiting for application startup.
INFO: Application startup complete."><svg aria-hidden="true" class="octicon octicon-copy js-clipboard-copy-icon" data-view-component="true" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><span style="background-color: white;"><path d="M0 6.75C0 5.784.784 5 1.75 5h1.5a.75.75 0 0 1 0 1.5h-1.5a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-1.5a.75.75 0 0 1 1.5 0v1.5A1.75 1.75 0 0 1 9.25 16h-7.5A1.75 1.75 0 0 1 0 14.25Z"></path><path d="M5 1.75C5 .784 5.784 0 6.75 0h7.5C15.216 0 16 .784 16 1.75v7.5A1.75 1.75 0 0 1 14.25 11h-7.5A1.75 1.75 0 0 1 5 9.25Zm1.75-.25a.25.25 0 0 0-.25.25v7.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-7.5a.25.25 0 0 0-.25-.25Z"></path></span></span></span></clipboard-copy></div></div><h2 dir="auto" style="border-bottom: 1px solid var(--borderColor-muted, var(--color-border-muted)); box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;" tabindex="-1"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/OpenAI-Whatsapp-Bot?tab=readme-ov-file#3-setup-ngrok" id="user-content-3-setup-ngrok" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; position: absolute; text-decoration-line: none; text-underline-offset: 0.2rem;" tabindex="-1"><span><span style="background-color: white; color: black;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></span></span></span></span></a><span style="background-color: white;">3) Setup Ngrok</span></h2><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">Go to Ngrok and download the utility (<a href="https://ngrok.com/download" rel="nofollow" style="box-sizing: border-box; text-underline-offset: 0.2rem;">https://ngrok.com/download</a>) then just expose your port 8000 to ngrok via <code style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em; white-space-collapse: break-spaces;">ngrok http http://localhost:8000</code> if you don't have a domain. Ngrok by default will provide a random domain e.g. <code style="border-radius: 6px; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em; white-space-collapse: break-spaces;"> https://621c00345df3.ngrok.app</code></span></p><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">Navigate to your Twilio account and enter the end-point in the sandbox configuration:</span></p><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjXdyY4tKKKLSHBCUSIbMiYv6gTOGfOiokIs4GtpOAWYN3C9ouVHWdlD8BLO7l4ty4TBOfJsKh20yiTC0_RHmtRPbu-KWoYqFuULpcsv1UbBxnBLbZ3kWHwUZUKuMsm8_GpMnZPDhltKPO3K_3LoaTCQrrcRA0H2IInTL2cTh_S3zlXkaRaHf0SZJn_XAs" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="562" data-original-width="966" height="372" src="https://blogger.googleusercontent.com/img/a/AVvXsEjXdyY4tKKKLSHBCUSIbMiYv6gTOGfOiokIs4GtpOAWYN3C9ouVHWdlD8BLO7l4ty4TBOfJsKh20yiTC0_RHmtRPbu-KWoYqFuULpcsv1UbBxnBLbZ3kWHwUZUKuMsm8_GpMnZPDhltKPO3K_3LoaTCQrrcRA0H2IInTL2cTh_S3zlXkaRaHf0SZJn_XAs=w640-h372" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjrrF5NyxAuIu6pguPk1ZAh1M-cI3jIhQKKV91C2dkOh-PKWWi2bSJYgCWqp8MoheADCoRo_NLuApg5kdp4HTuZQhYpcgwADpMi4o_D663idIMIcws7Db2iMgMSHfbGkCNnDiffq2BJgBXAjGGz8afWzQX1mQwiX9sG7I4iWI5nG3JK6TLSX_3KKlYzZXM" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="91" data-original-width="246" height="118" src="https://blogger.googleusercontent.com/img/a/AVvXsEjrrF5NyxAuIu6pguPk1ZAh1M-cI3jIhQKKV91C2dkOh-PKWWi2bSJYgCWqp8MoheADCoRo_NLuApg5kdp4HTuZQhYpcgwADpMi4o_D663idIMIcws7Db2iMgMSHfbGkCNnDiffq2BJgBXAjGGz8afWzQX1mQwiX9sG7I4iWI5nG3JK6TLSX_3KKlYzZXM" width="320" /></a></div><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><br /></p><h2 dir="auto" style="border-bottom: 1px solid var(--borderColor-muted, var(--color-border-muted)); box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;" tabindex="-1"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/OpenAI-Whatsapp-Bot?tab=readme-ov-file#4-python-api-interaction" id="user-content-4-python-api-interaction" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; position: absolute; text-decoration-line: none; text-underline-offset: 0.2rem;" tabindex="-1"><span><span style="background-color: white; color: black;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></span></span></span></span></a><span style="background-color: white;">4) Python API interaction</span></h2><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">After running the bot.py and setting up the twilio account and interacting with whatsapp, you'll be able to see the activity on your screen:</span></p><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://private-user-images.githubusercontent.com/7347994/298407327-9374ef31-880d-46a0-89cb-09b3b7ebfa54.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDc2NzI0NDgsIm5iZiI6MTcwNzY3MjE0OCwicGF0aCI6Ii83MzQ3OTk0LzI5ODQwNzMyNy05Mzc0ZWYzMS04ODBkLTQ2YTAtODljYi0wOWIzYjdlYmZhNTQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIxMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMTFUMTcyMjI4WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OGI5ZDMwMDdkZTM2OTQwM2Q2MTIyMzI1ZmM3ODYzNDBlNzBlZjNkMWYyYTlkMDNiN2U1OGUwOTBmMmFhMmRlNiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.YWzBx89hGLJ7X8VtquMyL-zaJWBi9Ib89dy5OjaC5Eo" rel="noopener noreferrer" style="box-sizing: border-box; text-underline-offset: 0.2rem;" target="_blank"><span style="background-color: white; color: black;"><img alt="image" src="https://private-user-images.githubusercontent.com/7347994/298407327-9374ef31-880d-46a0-89cb-09b3b7ebfa54.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDc2NzI0NDgsIm5iZiI6MTcwNzY3MjE0OCwicGF0aCI6Ii83MzQ3OTk0LzI5ODQwNzMyNy05Mzc0ZWYzMS04ODBkLTQ2YTAtODljYi0wOWIzYjdlYmZhNTQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDIxMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAyMTFUMTcyMjI4WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OGI5ZDMwMDdkZTM2OTQwM2Q2MTIyMzI1ZmM3ODYzNDBlNzBlZjNkMWYyYTlkMDNiN2U1OGUwOTBmMmFhMmRlNiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.YWzBx89hGLJ7X8VtquMyL-zaJWBi9Ib89dy5OjaC5Eo" style="border-style: none; box-sizing: content-box; max-width: 100%;" /></span></a></p><h2 dir="auto" style="border-bottom: 1px solid var(--borderColor-muted, var(--color-border-muted)); box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;" tabindex="-1"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/OpenAI-Whatsapp-Bot?tab=readme-ov-file#5-database-message-storage" id="user-content-5-database-message-storage" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; position: absolute; text-decoration-line: none; text-underline-offset: 0.2rem;" tabindex="-1"><span><span style="background-color: white; color: black;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></span></span></span></span></a><span style="background-color: white;">5) Database Message Storage</span></h2><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><span style="background-color: white;">As each message gets</span></p><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgwVH2kexlTs8qj6qOGDgBiocSdQn6rnQDayLLhcS_dJSasQurieGjcbLUzk7XJUAlp4_XxYH44BDcKnoz8vgKxzu8b5XRZVJBC0-8ZqggRYEMp1cz2yNZgxUz59U_qdDwuy0eWxtYOG8Gs_6KqbR9LOplhLp7F0PjW0sRSX1Tcto8K4SRqno-q-GOQuVM" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="402" data-original-width="1293" height="198" src="https://blogger.googleusercontent.com/img/a/AVvXsEgwVH2kexlTs8qj6qOGDgBiocSdQn6rnQDayLLhcS_dJSasQurieGjcbLUzk7XJUAlp4_XxYH44BDcKnoz8vgKxzu8b5XRZVJBC0-8ZqggRYEMp1cz2yNZgxUz59U_qdDwuy0eWxtYOG8Gs_6KqbR9LOplhLp7F0PjW0sRSX1Tcto8K4SRqno-q-GOQuVM=w640-h198" width="640" /></a></div><br /><br /><p></p><h2 dir="auto" style="border-bottom: 1px solid var(--borderColor-muted, var(--color-border-muted)); box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-weight: var(--base-text-weight-semibold, 600); line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;" tabindex="-1"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/OpenAI-Whatsapp-Bot?tab=readme-ov-file#6-final-result-talking-through-whatsapp" id="user-content-6-final-result-talking-through-whatsapp" style="box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; position: absolute; text-decoration-line: none; text-underline-offset: 0.2rem;" tabindex="-1"><span><span style="background-color: white; color: black;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"></svg><span><span><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></span></span></span></span></a><span style="background-color: white;">6) Final Result, talking through WhatsApp</span></h2><p dir="auto" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 0px; margin-top: 0px;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhLz1QmaBKzvTa1zJVqaTjYwKjIg_djeJt55GMyhncbxT85R2icZAAloT59Q2vS7qYbZHT_RLqLLzCFzB4VSHXnTN2zBpys6zeuq4XQG2Q8q8Q9VdZYrywalI7D8fb4Nu3o5_lQHV6uccJ6QA62W22ZV71IHorwtijNfGxlD35vwtKPdZmZrls16clD-VI" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="640" data-original-width="539" height="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEhLz1QmaBKzvTa1zJVqaTjYwKjIg_djeJt55GMyhncbxT85R2icZAAloT59Q2vS7qYbZHT_RLqLLzCFzB4VSHXnTN2zBpys6zeuq4XQG2Q8q8Q9VdZYrywalI7D8fb4Nu3o5_lQHV6uccJ6QA62W22ZV71IHorwtijNfGxlD35vwtKPdZmZrls16clD-VI=w539-h640" width="539" /></a></div><br /><br /><p></p>Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-71497036426954882072021-04-05T21:00:00.003+02:002024-02-11T19:40:34.081+01:00Web scraping with Python 3.7, Docker, and Kubernetes<p><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; color: #24292e; font-size: 16px;"></span></p><div class="separator" style="clear: both; text-align: center;"><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; color: #24292e; font-size: 16px;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrSFJnr2UBoH6LWj64Jbst0S1HMk288l1YMW8zia8vZNpA4goytX75cZoeJC78ft26JIqL2djpeVYg3b6_ReFzhWnRZHQ0REC1IkwkByG9LWaFtM_GnsBvESgtRLafBWvJ8f5izD3lzOSrOSousKW1qXD56w564_Xz4hX81FtkjGnHFYg-0Cp7UUJqMpk/s1024/DALL%C2%B7E%202024-02-11%2018.39.57%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_Web%20Scraping%20with%20Python%203.7,%20Docker,%20and%20Kubernetes_.%20The%20image%20should%20visually%20represent%20the%20integr.webp" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1024" data-original-width="1024" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrSFJnr2UBoH6LWj64Jbst0S1HMk288l1YMW8zia8vZNpA4goytX75cZoeJC78ft26JIqL2djpeVYg3b6_ReFzhWnRZHQ0REC1IkwkByG9LWaFtM_GnsBvESgtRLafBWvJ8f5izD3lzOSrOSousKW1qXD56w564_Xz4hX81FtkjGnHFYg-0Cp7UUJqMpk/w200-h200/DALL%C2%B7E%202024-02-11%2018.39.57%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_Web%20Scraping%20with%20Python%203.7,%20Docker,%20and%20Kubernetes_.%20The%20image%20should%20visually%20represent%20the%20integr.webp" width="200" /></a></span></div><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; color: #24292e; font-size: 16px;">Web scraping scripts to extract financial data. In a nutshell, this method can help you to get any information that it's available on any website using the </span><a href="https://www.crummy.com/software/BeautifulSoup/" rel="nofollow" style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; text-decoration-line: none;">BeautifulSoup</a><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; color: #24292e; font-size: 16px;"> library and python. The idea is to use this library to parse any DOM and get the data that we are interested in. Getting data from a list for example is a very simple job. Things get more interesting when we want to download more complex data like a table.</span><p></p><p><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; color: #24292e; font-size: 16px;"><br /></span></p><p><span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; color: #24292e; font-size: 16px;"><br /></span></p><h2 style="background-color: white; border-bottom: 1px solid var(--color-border-secondary); box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/web-scraping/blob/master/README.md#1-installing-beautifulsoup" id="user-content-1-installing-beautifulsoup" style="background-color: initial; box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>1) Installing BeautifulSoup</h2><div class="highlight highlight-source-batchfile" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">pip install beautifulsoup4
or
python -m pip install beautifulsoup4</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Output:</p><div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">python -m pip install beautifulsoup4
Collecting beautifulsoup4
Downloading beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">|</span>████████████████████████████████<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">|</span> 115 kB 3.3 MB/s
Collecting soupsieve<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span>1.2<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">;</span> python_version <span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span>= <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>3.0<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>
Downloading soupsieve-2.2-py3-none-any.whl (33 kB)
Installing collected packages: soupsieve, beautifulsoup4
Successfully installed beautifulsoup4-4.9.3 soupsieve-2.2</pre></div><h2 style="background-color: white; border-bottom: 1px solid var(--color-border-secondary); box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/web-scraping/blob/master/README.md#2-simple-example-scraping-names" id="user-content-2-simple-example-scraping-names" style="background-color: initial; box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>2) Simple Example (scraping names)</h2><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">The following script example tries to generate random user names via web scraping. Firstly, we locate a website that contains a list of name that we can download and then we use this list to generate user names. To do this, we can browse for any of the top 1000 girl names and see any of the links available:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">In our case, we found <a href="https://family.disney.com/articles/1000-most-popular-girl-names/" rel="nofollow" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;">this</a> url which seems pretty good for what we need. Upon inspecting the names, we can see that the DOM is pretty straightforward and each name is placed under 'li' tags and the whole group under an 'ol' tag:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/web-scraping/blob/master/baby-names.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/web-scraping/raw/master/baby-names.png" style="background-color: var(--color-bg-primary); border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">To get the list of names on a usable format using python, we can use the BeautifulSoup library and locate the specific tags we want (ol and then li, and print the content of it). The code below shows how this can be done:</p><div class="highlight highlight-source-python" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Let's create a random user name generator using web scraping.</span>
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># first we need to download a list of names to work on.</span>
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># the following url contains 1000 girl names and we will download them using web scraping.</span>
<span class="pl-s1" style="box-sizing: border-box;">url</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'https://family.disney.com/articles/1000-most-popular-girl-names/'</span>
<span class="pl-s1" style="box-sizing: border-box;">page</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">requests</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">get</span>(<span class="pl-s1" style="box-sizing: border-box;">url</span>)
<span class="pl-s1" style="box-sizing: border-box;">content</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">page</span>.<span class="pl-s1" style="box-sizing: border-box;">content</span>
<span class="pl-s1" style="box-sizing: border-box;">soup</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-v" color="var(--color-prettylights-syntax-variable)" style="box-sizing: border-box;">BeautifulSoup</span>(<span class="pl-s1" style="box-sizing: border-box;">content</span>, <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'html.parser'</span>)
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Find the first 'ol' that contains the list of items</span>
<span class="pl-s1" style="box-sizing: border-box;">list</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">soup</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">find</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'ol'</span>)
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Each name in the list is stored under a 'li' item</span>
<span class="pl-s1" style="box-sizing: border-box;">list_names</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">list</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">find_all</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'li'</span>)
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Now print all the names that we have scraped from the web site</span>
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">for</span> <span class="pl-s1" style="box-sizing: border-box;">name</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">in</span> <span class="pl-s1" style="box-sizing: border-box;">list_names</span>:
<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">print</span>(<span class="pl-s1" style="box-sizing: border-box;">name</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">get_text</span>())</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/web-scraping/blob/master/list-names-scraping.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/web-scraping/raw/master/list-names-scraping.png" style="background-color: var(--color-bg-primary); border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">With a little tweak, we can easily generate usernames based on this data or whatever you want to do with it.</p><div class="highlight highlight-source-python" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Generate the following sequence </span>
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!()@$^&*[]</span>
<span class="pl-s1" style="box-sizing: border-box;">chars</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">string</span>.<span class="pl-s1" style="box-sizing: border-box;">ascii_letters</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">+</span> <span class="pl-s1" style="box-sizing: border-box;">string</span>.<span class="pl-s1" style="box-sizing: border-box;">digits</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">+</span> <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'!()@$^&*[]'</span>
<span class="pl-s1" style="box-sizing: border-box;">random</span>.<span class="pl-s1" style="box-sizing: border-box;">seed</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> (<span class="pl-s1" style="box-sizing: border-box;">os</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">urandom</span>(<span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">1024</span>))
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">for</span> <span class="pl-s1" style="box-sizing: border-box;">name</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">in</span> <span class="pl-s1" style="box-sizing: border-box;">list_names</span>:
<span class="pl-s1" style="box-sizing: border-box;">extra_digits</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">''</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">join</span>(<span class="pl-s1" style="box-sizing: border-box;">random</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">choice</span>(<span class="pl-s1" style="box-sizing: border-box;">string</span>.<span class="pl-s1" style="box-sizing: border-box;">digits</span>))
<span class="pl-s1" style="box-sizing: border-box;">extra_chars</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">''</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">join</span>(<span class="pl-s1" style="box-sizing: border-box;">random</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">choice</span>(<span class="pl-s1" style="box-sizing: border-box;">chars</span>) <span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">for</span> <span class="pl-s1" style="box-sizing: border-box;">i</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">in</span> <span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">range</span>(<span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">8</span>))
<span class="pl-s1" style="box-sizing: border-box;">username</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">name</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">get_text</span>().<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">lower</span>() <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">+</span> <span class="pl-s1" style="box-sizing: border-box;">extra_digits</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">+</span> <span class="pl-s1" style="box-sizing: border-box;">extra_chars</span>
<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">print</span>(<span class="pl-s1" style="box-sizing: border-box;">username</span>)</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/web-scraping/blob/master/list-usernames-scraping.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/web-scraping/raw/master/list-usernames-scraping.png" style="background-color: var(--color-bg-primary); border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><h2 style="background-color: white; border-bottom: 1px solid var(--color-border-secondary); box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/web-scraping/blob/master/README.md#3-complex-example-scraping-financial-information" id="user-content-3-complex-example-scraping-financial-information" style="background-color: initial; box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>3) Complex Example (scraping financial information)</h2><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">One of the most interesting uses for this technology is the ability to download large amounts of data that are table-based. This example tries to download the balance sheet from one of the stocks in Yahoo Finance. Imagine that we want to download the balance sheet of TSLA (if you want to download the data, you need to become a premium subscriber and they have made it difficult to perform web scraping). To perform this operation, we need to look at the way the table is created (a bunch of div tags) and how each row is composed (classes, ids) so they are easily identifiable.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Balance Sheet: <a href="https://finance.yahoo.com/quote/TSLA/balance-sheet?p=TSLA&_guc_consent_skip=1596652371" rel="nofollow" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;">https://finance.yahoo.com/quote/TSLA/balance-sheet?p=TSLA&_guc_consent_skip=1596652371</a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/web-scraping/blob/master/balancesheet.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/web-scraping/raw/master/balancesheet.png" style="background-color: var(--color-bg-primary); border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><div class="highlight highlight-source-python" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">import</span> <span class="pl-s1" style="box-sizing: border-box;">requests</span>
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">from</span> <span class="pl-s1" style="box-sizing: border-box;">bs4</span> <span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">import</span> <span class="pl-v" color="var(--color-prettylights-syntax-variable)" style="box-sizing: border-box;">BeautifulSoup</span>
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Download Balance Sheet table from TSLA</span>
<span class="pl-s1" style="box-sizing: border-box;">url</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'https://finance.yahoo.com/quote/TSLA/balance-sheet?p=TSLA'</span>
<span class="pl-s1" style="box-sizing: border-box;">page</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">requests</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">get</span>(<span class="pl-s1" style="box-sizing: border-box;">url</span>)
<span class="pl-s1" style="box-sizing: border-box;">content</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">page</span>.<span class="pl-s1" style="box-sizing: border-box;">content</span>
<span class="pl-s1" style="box-sizing: border-box;">soup</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-v" color="var(--color-prettylights-syntax-variable)" style="box-sizing: border-box;">BeautifulSoup</span>(<span class="pl-s1" style="box-sizing: border-box;">content</span>, <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'html.parser'</span>)
<span class="pl-s1" style="box-sizing: border-box;">cash_balance</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> {}
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Search for the main DIV that encloses the balance sheet table</span>
<span class="pl-s1" style="box-sizing: border-box;">main_content</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">soup</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">find_all</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'div'</span>, <span class="pl-s1" style="box-sizing: border-box;">class_</span><span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span><span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'M(0) Whs(n) BdEnd Bdc($seperatorColor) D(itb)'</span>)
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">for</span> <span class="pl-s1" style="box-sizing: border-box;">div</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">in</span> <span class="pl-s1" style="box-sizing: border-box;">main_content</span>:
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Look for each DIV that encloses every single row</span>
<span class="pl-s1" style="box-sizing: border-box;">sub_div</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">div</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">find_all</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'div'</span>, <span class="pl-s1" style="box-sizing: border-box;">class_</span><span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span><span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'D(tbr) fi-row Bgc($hoverBgColor):h'</span>)
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">for</span> <span class="pl-s1" style="box-sizing: border-box;">sub</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">in</span> <span class="pl-s1" style="box-sizing: border-box;">sub_div</span>:
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Select the first column as the index of our dictionary and select the second column as the data to store (2019)</span>
<span class="pl-s1" style="box-sizing: border-box;">cash_balance</span>[<span class="pl-s1" style="box-sizing: border-box;">sub</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">get_text</span>(<span class="pl-s1" style="box-sizing: border-box;">separator</span><span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span><span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"|"</span>).<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">split</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"|"</span>)[<span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">0</span>]] <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">sub</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">get_text</span>(<span class="pl-s1" style="box-sizing: border-box;">separator</span><span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span><span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"|"</span>).<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">split</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"|"</span>)[<span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">1</span>]
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;">#print(sub.get_text())</span></pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">The final result of the execution of the code above lets us produce the desired output, scraping the data from the Yahoo Finance page for the TSLA ticker:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/web-scraping/blob/master/balancesheetdictionary.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/web-scraping/raw/master/balancesheetdictionary.png" style="background-color: var(--color-bg-primary); border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><h1 style="background-color: white; border-bottom: 1px solid var(--color-border-secondary); box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin: 24px 0px 16px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/web-scraping/blob/master/README.md#containeraizing-the-script-with-docker-and-kubernetes" id="user-content-containeraizing-the-script-with-docker-and-kubernetes" style="background-color: initial; box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>Containerizing the script with Docker and Kubernetes</h1><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">In order to make the script easily deployable, we'll create a Flask service that will host the retrieval of the cash balances and it will be all contained in a docker image.</p><h2 style="background-color: white; border-bottom: 1px solid var(--color-border-secondary); box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/web-scraping/blob/master/README.md#1-create-the-flask-service" id="user-content-1-create-the-flask-service" style="background-color: initial; box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>1) Create the Flask service</h2><div class="highlight highlight-source-python" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">import</span> <span class="pl-s1" style="box-sizing: border-box;">requests</span>
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">from</span> <span class="pl-s1" style="box-sizing: border-box;">bs4</span> <span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">import</span> <span class="pl-v" color="var(--color-prettylights-syntax-variable)" style="box-sizing: border-box;">BeautifulSoup</span>
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">from</span> <span class="pl-s1" style="box-sizing: border-box;">flask</span> <span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">import</span> <span class="pl-v" color="var(--color-prettylights-syntax-variable)" style="box-sizing: border-box;">Flask</span>, <span class="pl-s1" style="box-sizing: border-box;">jsonify</span>
<span class="pl-s1" style="box-sizing: border-box;">server</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-v" color="var(--color-prettylights-syntax-variable)" style="box-sizing: border-box;">Flask</span>(<span class="pl-s1" style="box-sizing: border-box;">__name__</span>)
<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">@<span class="pl-s1" style="box-sizing: border-box;">server</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">route</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"/"</span>)</span>
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">def</span> <span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">cash_balance_get</span>():
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Download Balance Sheet table from TSLA</span>
<span class="pl-s1" style="box-sizing: border-box;">url</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'https://finance.yahoo.com/quote/TSLA/balance-sheet?p=TSLA'</span>
<span class="pl-s1" style="box-sizing: border-box;">page</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">requests</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">get</span>(<span class="pl-s1" style="box-sizing: border-box;">url</span>)
<span class="pl-s1" style="box-sizing: border-box;">content</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">page</span>.<span class="pl-s1" style="box-sizing: border-box;">content</span>
<span class="pl-s1" style="box-sizing: border-box;">soup</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-v" color="var(--color-prettylights-syntax-variable)" style="box-sizing: border-box;">BeautifulSoup</span>(<span class="pl-s1" style="box-sizing: border-box;">content</span>, <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'html.parser'</span>)
<span class="pl-s1" style="box-sizing: border-box;">cash_balance</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> {}
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Search for the main DIV that encloses the balance sheet table</span>
<span class="pl-s1" style="box-sizing: border-box;">main_content</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">soup</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">find_all</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'div'</span>, <span class="pl-s1" style="box-sizing: border-box;">class_</span><span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span><span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'M(0) Whs(n) BdEnd Bdc($seperatorColor) D(itb)'</span>)
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">for</span> <span class="pl-s1" style="box-sizing: border-box;">div</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">in</span> <span class="pl-s1" style="box-sizing: border-box;">main_content</span>:
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Look for each DIV that encloses every single row</span>
<span class="pl-s1" style="box-sizing: border-box;">sub_div</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">div</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">find_all</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'div'</span>, <span class="pl-s1" style="box-sizing: border-box;">class_</span><span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span><span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'D(tbr) fi-row Bgc($hoverBgColor):h'</span>)
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">for</span> <span class="pl-s1" style="box-sizing: border-box;">sub</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">in</span> <span class="pl-s1" style="box-sizing: border-box;">sub_div</span>:
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;"># Select the first column as the index of our dictionary and select the second column as the data to store (2019)</span>
<span class="pl-s1" style="box-sizing: border-box;">cash_balance</span>[<span class="pl-s1" style="box-sizing: border-box;">sub</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">get_text</span>(<span class="pl-s1" style="box-sizing: border-box;">separator</span><span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span><span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"|"</span>).<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">split</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"|"</span>)[<span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">0</span>]] <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span> <span class="pl-s1" style="box-sizing: border-box;">sub</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">get_text</span>(<span class="pl-s1" style="box-sizing: border-box;">separator</span><span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span><span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"|"</span>).<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">split</span>(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"|"</span>)[<span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">1</span>]
<span class="pl-c" color="var(--color-prettylights-syntax-comment)" style="box-sizing: border-box;">#print(sub.get_text())</span>
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">return</span> <span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">jsonify</span>(<span class="pl-s1" style="box-sizing: border-box;">cash_balance</span>)
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">if</span> <span class="pl-s1" style="box-sizing: border-box;">__name__</span> <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">==</span> <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"__main__"</span>:
<span class="pl-s1" style="box-sizing: border-box;">server</span>.<span class="pl-en" color="var(--color-prettylights-syntax-entity)" style="box-sizing: border-box;">run</span>(<span class="pl-s1" style="box-sizing: border-box;">host</span><span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">=</span><span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'0.0.0.0'</span>)</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">If we run this code and try to get to <code style="background-color: var(--color-markdown-code-bg); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">http://localhost:5000</code>, we'll get the following response:</p><div class="highlight highlight-source-json" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">{<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Capital Lease Obligations<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>1,540,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Common Stock Equity<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>22,225,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Invested Capital<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>33,964,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Net Debt<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>-<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Net Tangible Assets<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>21,705,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Ordinary Shares Number<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>960,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Share Issued<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>960,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Tangible Book Value<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>21,705,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Total Assets<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>52,148,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Total Capitalization<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>31,832,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Total Debt<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>13,279,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Total Equity Gross Minority Interest<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>23,075,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Total Liabilities Net Minority Interest<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>29,073,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>,<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>Working Capital<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>:<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>12,469,000<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>}</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">The output of the execution can be seen below:</p><div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">runfile(<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'</span>C:/Users/thund/Source/Repos/web-scraping/web_scraping_yahoo_finance_balance_sheet_server.py<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'</span></span>, wdir=<span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'</span>C:/Users/thund/Source/Repos/web-scraping<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'</span></span>)
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">*</span> Serving Flask app <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>web_scraping_yahoo_finance_balance_sheet_server<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span> (lazy loading)
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">*</span> Environment: production
WARNING: This is a development server. Do not use it <span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">in</span> a production deployment.
Use a production WSGI server instead.
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">*</span> Debug mode: off
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">*</span> Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [03/Apr/2021 16:00:03] <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>GET / HTTP/1.1<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span> 200 -
127.0.0.1 - - [03/Apr/2021 16:00:03] <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>GET /favicon.ico HTTP/1.1<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span> 404 -</pre></div><h2 style="background-color: white; border-bottom: 1px solid var(--color-border-secondary); box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/web-scraping/blob/master/README.md#2-preparing-the-docker-file" id="user-content-2-preparing-the-docker-file" style="background-color: initial; box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>2) Preparing the docker file</h2><pre lang="docker" style="border-radius: 6px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 16px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 6px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow-wrap: normal; overflow: visible; padding: 0px; word-break: normal;"># set base image (host OS)
FROM python:3.7
# set the working directory in the container
WORKDIR /code
# copy the dependencies file to the working directory
COPY web_scraping_packages.txt .
# install dependencies
RUN pip install -r web_scraping_packages.txt
# copy the script to the working directory
COPY web_scraping_yahoo_finance_balance_sheet_server.py .
# command to run on container start
CMD [ "python", "./web_scraping_yahoo_finance_balance_sheet_server.py" ]
</code></pre><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Include the packages to be installed as part of the docker image creation (<code style="background-color: var(--color-markdown-code-bg); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">web_scraping_packages.txt</code>):</p><div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">requests==2.22.0
beautifulsoup4==4.7.1
Flask==1.1.1</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Create the image:</p><div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">C:<span class="pl-cce" style="box-sizing: border-box;">\S</span>ource<span class="pl-cce" style="box-sizing: border-box;">\R</span>epos<span class="pl-cce" style="box-sizing: border-box;">\w</span>eb-scraping<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span>docker build -t web-scraping:v1 <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">.</span>
[+] Building 547.8s (10/10) FINISHED
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> [internal] load build definition from Dockerfile 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> transferring dockerfile: 526B 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> [internal] load .dockerignore 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> transferring context: 2B 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> [internal] load metadata <span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">for</span> docker.io/library/python:3.7 42.6s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> [1/5] FROM docker.io/library/python:3.7@sha256:8b3d5242ba72ac32785362c4ade75b61ce941bd454f8e1c585270736c991 495.1s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> resolve docker.io/library/python:3.7@sha256:8b3d5242ba72ac32785362c4ade75b61ce941bd454f8e1c585270736c9916f 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:8b3d5242ba72ac32785362c4ade75b61ce941bd454f8e1c585270736c9916fe6 1.86kB / 1.86kB 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:581b9e21ee31fd842281d787d150f92ff8850be52ab00ba7ad75f3167cb397a4 9.04kB / 9.04kB 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:48c2faf66abec3dce9f54d6722ff592fce6dd4fb58a0d0b72282936c6598a3b3 10.00MB / 10.00MB 50.9s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:b161a84f5d6d6a1588a3fbea618c464e0599094524a09b689cb3f455fbf33344 2.22kB / 2.22kB 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:004f1eed87df3f75f5e2a1a649fa7edd7f713d1300532fd0909bb39cd48437d7 50.43MB / 50.43MB 232.4s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:5d6f1e8117dbb1c6a57603cb4f321a861a08105a81bcc6b01b0ec2b78c8523a5 7.83MB / 7.83MB 23.8s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:234b70d0479d7f16d7ee8d04e4ffdacc57d7d14313faf59d332f18b2e9418743 51.84MB / 51.84MB 263.6s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:6fa07a00e2f029c4b2c7f177a2b696f1b3510040cde4f5bb06ddbca98e7fbf76 192.35MB / 192.35MB 479.7s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:04a31b4508b8e95fb3cb25486c4068185054895b12e0611e386a002ee9c0e07c 6.15MB / 6.15MB 268.2s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> extracting sha256:004f1eed87df3f75f5e2a1a649fa7edd7f713d1300532fd0909bb39cd48437d7 7.7s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> extracting sha256:5d6f1e8117dbb1c6a57603cb4f321a861a08105a81bcc6b01b0ec2b78c8523a5 0.9s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> extracting sha256:48c2faf66abec3dce9f54d6722ff592fce6dd4fb58a0d0b72282936c6598a3b3 0.9s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:9039bc4ee433b9b3917b5a7506b20c4335921059058fb626f21c10942f68bf1d 16.33MB / 16.33MB 326.7s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> extracting sha256:234b70d0479d7f16d7ee8d04e4ffdacc57d7d14313faf59d332f18b2e9418743 5.3s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:9d4af2d5ba9c3e8e735b90ee7041e6bb0d8a21d8e816a0ed781c4feca39d6da1 233B / 233B 276.2s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> sha256:dc46d185ff1dbb7fd44a2d045b44e3c8edeec432507a2c7712ff6d31bf802aec 2.17MB / 2.17MB 284.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> extracting sha256:6fa07a00e2f029c4b2c7f177a2b696f1b3510040cde4f5bb06ddbca98e7fbf76 12.2s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> extracting sha256:04a31b4508b8e95fb3cb25486c4068185054895b12e0611e386a002ee9c0e07c 0.5s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> extracting sha256:9039bc4ee433b9b3917b5a7506b20c4335921059058fb626f21c10942f68bf1d 1.1s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> extracting sha256:9d4af2d5ba9c3e8e735b90ee7041e6bb0d8a21d8e816a0ed781c4feca39d6da1 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> extracting sha256:dc46d185ff1dbb7fd44a2d045b44e3c8edeec432507a2c7712ff6d31bf802aec 0.3s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> [internal] load build context 0.1s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> transferring context: 2.12kB 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> [2/5] WORKDIR /code 1.4s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> [3/5] COPY web_scraping_packages.txt <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">.</span> 0.1s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> [4/5] RUN pip install -r web_scraping_packages.txt 7.9s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> [5/5] COPY web_scraping_yahoo_finance_balance_sheet_server.py <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">.</span> 0.1s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> exporting to image 0.3s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> exporting layers 0.3s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> writing image sha256:d36f971bc093676c0d91b95308e1a3097aff29de5b55c4cb912e7903cc60b49e 0.0s
=<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> =<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> naming to docker.io/library/web-scraping:v1 0.0s</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://user-images.githubusercontent.com/7347994/113489653-5cb49380-94bd-11eb-96c6-efcfac1c8812.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;" target="_blank"><img alt="image" src="https://user-images.githubusercontent.com/7347994/113489653-5cb49380-94bd-11eb-96c6-efcfac1c8812.png" style="background-color: var(--color-bg-primary); border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><h2 style="background-color: white; border-bottom: 1px solid var(--color-border-secondary); box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/web-scraping/blob/master/README.md#3-running-the-docker-image" id="user-content-3-running-the-docker-image" style="background-color: initial; box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>3) Running the docker image</h2><div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">C:<span class="pl-cce" style="box-sizing: border-box;">\W</span>INDOWS<span class="pl-cce" style="box-sizing: border-box;">\s</span>ystem<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">32></span>docker run -it --rm -p 5000:5000 web-scraping:v1
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">*</span> Serving Flask app <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>web_scraping_yahoo_finance_balance_sheet_server<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span> (lazy loading)
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">*</span> Environment: production
WARNING: This is a development server. Do not use it <span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">in</span> a production deployment.
Use a production WSGI server instead.
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">*</span> Debug mode: off
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">*</span> Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
172.17.0.1 - - [03/Apr/2021 17:14:13] <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>GET / HTTP/1.1<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span> 200 -</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">If you browse <code style="background-color: var(--color-markdown-code-bg); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">http://localhost:5000</code>, you'll get the content of it in json format.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Once you are done, If you don't want to use the image anymore, use the following to delete it:</p><div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">C:<span class="pl-cce" style="box-sizing: border-box;">\W</span>INDOWS<span class="pl-cce" style="box-sizing: border-box;">\s</span>ystem<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">32></span>docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
web-scraping v1 791b2aef33a3 19 minutes ago 890MB
C:<span class="pl-cce" style="box-sizing: border-box;">\W</span>INDOWS<span class="pl-cce" style="box-sizing: border-box;">\s</span>ystem<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">32></span>docker rmi -f 791b2aef33a3
Untagged: web-scraping:v1
Deleted: sha256:791b2aef33a3641f7335102f7b8edeab2463b5e4eed046594cd5053c86a9c1f0</pre></div><h1 style="background-color: white; border-bottom: 1px solid var(--color-border-secondary); box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; line-height: 1.25; margin: 24px 0px 16px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/web-scraping/blob/master/README.md#running-your-docker-image-in-kubernetes" id="user-content-running-your-docker-image-in-kubernetes" style="background-color: initial; box-sizing: border-box; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>Running your Docker image in Kubernetes</h1><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">** Remember to enable Kubernetes if you are using Docker desktop on windows: <a href="https://user-images.githubusercontent.com/7347994/113489788-114eb500-94be-11eb-9448-32d60b80b266.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;" target="_blank"><img alt="image" src="https://user-images.githubusercontent.com/7347994/113489788-114eb500-94be-11eb-9448-32d60b80b266.png" style="background-color: var(--color-bg-primary); border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">To make things a bit interesting, we are going to deploy the docker image into a Kubernetes cluster. To do this, we will need to create the configuration files for the deployment and the service.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Deployment:</p><div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">apiVersion</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">apps/v1</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">kind</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">Deployment</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">metadata</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">name</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">web-scraping-deployment</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">spec</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">selector</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">matchLabels</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">app</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">web-scraping-pod</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">template</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">metadata</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">labels</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">app</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">web-scraping-pod</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">spec</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">containers</span>:
- <span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">name</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">web-scraping-container</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">image</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">web-scraping:v1</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">resources</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">limits</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">memory</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>128Mi<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">cpu</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;"><span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span>500m<span class="pl-pds" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">"</span></span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">ports</span>:
- <span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">containerPort</span>: <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">5000</span></pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Service:</p><div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">apiVersion</span>: <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">v1</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">kind</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">Service</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">metadata</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">name</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">web-scraping-service</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">spec</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">selector</span>:
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">app</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">web-scraping-pod</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">ports</span>:
- <span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">port</span>: <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">5000</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">targetPort</span>: <span class="pl-c1" color="var(--color-prettylights-syntax-constant)" style="box-sizing: border-box;">5000</span>
<span class="pl-ent" color="var(--color-prettylights-syntax-entity-tag)" style="box-sizing: border-box;">type</span>: <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">LoadBalancer</span></pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Then run the following:</p><div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">C:<span class="pl-cce" style="box-sizing: border-box;">\S</span>ource<span class="pl-cce" style="box-sizing: border-box;">\R</span>epos<span class="pl-cce" style="box-sizing: border-box;">\w</span>eb-scraping<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span>kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">*</span> docker-desktop docker-desktop docker-desktop
docker-for-desktop docker-desktop docker-desktop
minikube minikube minikube</pre></div><div class="highlight highlight-source-shell" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">C:<span class="pl-cce" style="box-sizing: border-box;">\U</span>sers<span class="pl-cce" style="box-sizing: border-box;">\t</span>hund<span class="pl-cce" style="box-sizing: border-box;">\S</span>ource<span class="pl-cce" style="box-sizing: border-box;">\R</span>epos<span class="pl-cce" style="box-sizing: border-box;">\w</span>eb-scraping<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span>kubectl apply -f .<span class="pl-cce" style="box-sizing: border-box;">\d</span>eployment.yml
deployment.apps/web-scraping-deployment created
C:<span class="pl-cce" style="box-sizing: border-box;">\U</span>sers<span class="pl-cce" style="box-sizing: border-box;">\t</span>hund<span class="pl-cce" style="box-sizing: border-box;">\S</span>ource<span class="pl-cce" style="box-sizing: border-box;">\R</span>epos<span class="pl-cce" style="box-sizing: border-box;">\w</span>eb-scraping<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span>kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
web-scraping-deployment 1/1 1 1 26s
C:<span class="pl-cce" style="box-sizing: border-box;">\U</span>sers<span class="pl-cce" style="box-sizing: border-box;">\t</span>hund<span class="pl-cce" style="box-sizing: border-box;">\S</span>ource<span class="pl-cce" style="box-sizing: border-box;">\R</span>epos<span class="pl-cce" style="box-sizing: border-box;">\w</span>eb-scraping<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span>kubectl apply -f .<span class="pl-cce" style="box-sizing: border-box;">\s</span>ervice.yml
service/web-scraping-service created
C:<span class="pl-cce" style="box-sizing: border-box;">\U</span>sers<span class="pl-cce" style="box-sizing: border-box;">\t</span>hund<span class="pl-cce" style="box-sizing: border-box;">\S</span>ource<span class="pl-cce" style="box-sizing: border-box;">\R</span>epos<span class="pl-cce" style="box-sizing: border-box;">\w</span>eb-scraping<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span>kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;"><</span>none<span class="pl-k" color="var(--color-prettylights-syntax-keyword)" style="box-sizing: border-box;">></span> 443/TCP 8h
web-scraping-service LoadBalancer 10.98.0.224 localhost 5000:32024/TCP 9s</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">You should be able to see the following containers running in Docker:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://user-images.githubusercontent.com/7347994/113612655-09bb1780-9648-11eb-94f6-6cd3f1211639.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;" target="_blank"><img alt="image" src="https://user-images.githubusercontent.com/7347994/113612655-09bb1780-9648-11eb-94f6-6cd3f1211639.png" style="background-color: var(--color-bg-primary); border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">After this, we will be able to see the content of it on <code style="background-color: var(--color-markdown-code-bg); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0.4em;">http://localhost:5000</code></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 0px; margin-top: 0px;"><a href="https://user-images.githubusercontent.com/7347994/113613036-89e17d00-9648-11eb-98b5-f52bd533d102.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; text-decoration-line: none;" target="_blank"><img alt="image" src="https://user-images.githubusercontent.com/7347994/113613036-89e17d00-9648-11eb-98b5-f52bd533d102.png" style="background-color: var(--color-bg-primary); border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 0px; margin-top: 0px;"><br /></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 0px; margin-top: 0px;">Source code with samples can be found here:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 0px; margin-top: 0px;">- https://github.com/JordiCorbilla/web-scraping </p>Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-40579001510531210042020-11-18T21:56:00.006+01:002024-02-11T23:23:42.528+01:00Hosting an ASP.NET Core 5.0 Web API in a Windows Service<p style="text-align: justify;"></p><p></p><p style="text-align: justify;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh66QnONGDqf-HXJTWGhPI82SsDHs3Ns1zUHE0c5wij_HMQsPQxeCRXyzhj_JF1VgB_IIvzhKxDE4-TK9fZxq22_0Z2jFfcL7i5SpzT9YOhBFh5nihyphenhyphenygONaFi-IGdkkHs5NNuvpJMMHdYQpO4_9COqw5vLoSDC2bJ9RkKeqCTWMovudRoo84hoQvqlMYM/s1024/DALL%C2%B7E%202024-02-11%2022.22.47%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_Hosting%20an%20ASP.NET%20Core%205.0%20Web%20API%20in%20a%20Windows%20Service_.%20The%20image%20should%20visually%20represent%20the%20co.webp" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1024" data-original-width="1024" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh66QnONGDqf-HXJTWGhPI82SsDHs3Ns1zUHE0c5wij_HMQsPQxeCRXyzhj_JF1VgB_IIvzhKxDE4-TK9fZxq22_0Z2jFfcL7i5SpzT9YOhBFh5nihyphenhyphenygONaFi-IGdkkHs5NNuvpJMMHdYQpO4_9COqw5vLoSDC2bJ9RkKeqCTWMovudRoo84hoQvqlMYM/w200-h200/DALL%C2%B7E%202024-02-11%2022.22.47%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_Hosting%20an%20ASP.NET%20Core%205.0%20Web%20API%20in%20a%20Windows%20Service_.%20The%20image%20should%20visually%20represent%20the%20co.webp" width="200" /></a></div>One of the coolest things I've seen with the .NET ecosystem is that you can host your Web API in a windows service. In this short article, you can find the source code for the template that I have built and that you can use to power your projects up.<p></p><p style="text-align: justify;">The template is very simple and it provides a basic skeleton for an ASP.NET Core 5.0 Web API project with hosted services and with the possibility to host the entire solution in a Windows Service. </p><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSmbaoscGQtZY_A5txqzS-C9gIholhdA2PjZh4EFQO3EkFSp4GKaLrSfw4qFkpypMsEJPkw_Q5qz3PpJGwsKRP2tIT6ZNprZ0OunVKY5kXZO5hnhWOfiNzzLsrUwSpCDA-VApQ8wre8bE/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="329" data-original-width="1404" height="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSmbaoscGQtZY_A5txqzS-C9gIholhdA2PjZh4EFQO3EkFSp4GKaLrSfw4qFkpypMsEJPkw_Q5qz3PpJGwsKRP2tIT6ZNprZ0OunVKY5kXZO5hnhWOfiNzzLsrUwSpCDA-VApQ8wre8bE/w640-h150/image.png" width="640" /></a></div><p></p><p>You can find the source code here: <a href="https://github.com/JordiCorbilla/WindowsServiceHost">https://github.com/JordiCorbilla/WindowsServiceHost</a></p><p>The project contains the basic configuration with the latest release available of .NET (at the time of writing this post) and how to easily operate with it and start working with it.</p><p>The idea behind the project is to be able to host the end-point without requiring IIS as you might have not installed it on your server and you want something a bit more powerful and scalable.</p><p>Let me know what you think and if you have further ideas to improve the template. This one is very basic and it uses ASP.NET Core 5.0 Web API, Worker services, HTTP/HTTPS, and the swagger UI with the following packages installed:</p><h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">Packages Used</h3><div class="highlight highlight-text-xml" style="background-color: white; box-sizing: border-box; color: #24292e; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: var(--color-bg-tertiary); border-radius: 6px; box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">PackageReference</span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Include</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Serilog<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Version</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>2.10.0<span class="pl-pds" style="box-sizing: border-box;">"</span></span> />
<<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">PackageReference</span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Include</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Swashbuckle.AspNetCore<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Version</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>5.6.3<span class="pl-pds" style="box-sizing: border-box;">"</span></span> />
<<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">PackageReference</span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Include</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Swashbuckle.AspNetCore.Annotations<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Version</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>5.6.3<span class="pl-pds" style="box-sizing: border-box;">"</span></span> />
<<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">PackageReference</span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Include</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Microsoft.Extensions.Hosting.WindowsServices<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Version</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>5.0.0<span class="pl-pds" style="box-sizing: border-box;">"</span></span> />
<<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">PackageReference</span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Include</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Serilog.AspNetCore<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Version</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>3.4.0<span class="pl-pds" style="box-sizing: border-box;">"</span></span> />
<<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">PackageReference</span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Include</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Serilog.Sinks.Console<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Version</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>3.1.1<span class="pl-pds" style="box-sizing: border-box;">"</span></span> />
<<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">PackageReference</span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Include</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Swashbuckle.AspNetCore.Swagger<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Version</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>5.6.3<span class="pl-pds" style="box-sizing: border-box;">"</span></span> />
<<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">PackageReference</span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Include</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Microsoft.AspNetCore.Mvc.NewtonsoftJson<span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-e" style="box-sizing: border-box; color: #6f42c1;">Version</span>=<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>5.0.0<span class="pl-pds" style="box-sizing: border-box;">"</span></span> /></pre></div><p>The important one to mention here is the "<span face="SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace" style="color: #032f62; font-size: 13.6px;">Microsoft.Extensions.Hosting.WindowsServices</span>" which is the one that will allow us to use the "<b>UseWindowsService()</b>" in the <b>IHostBuilder </b>interface.</p><p><b>Additional work to read:</b></p><p></p><ul style="text-align: left;"><li><a href="https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/windows-service?view=aspnetcore-5.0&tabs=visual-studio">https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/windows-service?view=aspnetcore-5.0&tabs=visual-studio</a></li><li><a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-5.0&tabs=visual-studio">https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-5.0&tabs=visual-studio</a></li><li><a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-5.0&tabs=visual-studio">https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-5.0&tabs=visual-studio</a></li></ul><p></p>Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com1tag:blogger.com,1999:blog-7022755517551007355.post-38598417062678875902020-08-09T21:05:00.010+02:002024-02-11T23:24:31.277+01:00GitHub Actions to publish .net core artifacts<p><span face="" style="background-color: white; color: #24292e; font-size: 16px;"></span></p><div class="separator" style="clear: both; text-align: center;"><span face="" style="background-color: white; color: #24292e; font-size: 16px;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb49cPNty4Wiyv6VemGtz5_wO9-9_ZF6K84Mr_893dKAg9XdPwasjSIdGPBlWV46nt_4gNe9KG623FOQ0CQnY9Zv0gM01W_UrHCH0-QVpWIurYgBegE-FR3-JT5mxbvspVIxARpBwnJxz4H_HLY7i1YmeJaayVTo9tUhV0JxyzygP9qiWWJLUMaUT9yN0/s1024/DALL%C2%B7E%202024-02-11%2022.23.51%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_GitHub%20Actions%20to%20Publish%20.NET%20Core%20Artifacts_.%20The%20image%20should%20visually%20represent%20the%20process%20of%20us.webp" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1024" data-original-width="1024" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb49cPNty4Wiyv6VemGtz5_wO9-9_ZF6K84Mr_893dKAg9XdPwasjSIdGPBlWV46nt_4gNe9KG623FOQ0CQnY9Zv0gM01W_UrHCH0-QVpWIurYgBegE-FR3-JT5mxbvspVIxARpBwnJxz4H_HLY7i1YmeJaayVTo9tUhV0JxyzygP9qiWWJLUMaUT9yN0/w200-h200/DALL%C2%B7E%202024-02-11%2022.23.51%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_GitHub%20Actions%20to%20Publish%20.NET%20Core%20Artifacts_.%20The%20image%20should%20visually%20represent%20the%20process%20of%20us.webp" width="200" /></a></span></div><span face="" style="background-color: white; color: #24292e; font-size: 16px;">In this article, you'll find how easy and simple it is to generate your .net core artifacts (binaries) using Github Actions.</span><p></p><h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><br /></h2><h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><br /></h2><h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><br /></h2><h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/github-actions/blob/master/README.md#adding-your-action" id="user-content-adding-your-action" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>Adding your action</h2><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Github offers a set of predefined actions that you can use to leverage the needs of your deployment machine. Maybe you need to generate artifacts for a Windows machine or Mac or Linux and you want to generate all of them without having to build things separately. The easiest way to use continuous integration through Github Actions is via a workflow configuration file. Go to Actions and click new workflow:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/github-actions/raw/master/githubactions.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/github-actions/raw/master/githubactions.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Select the netcore action to build your .net core application:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/github-actions/raw/master/netcoreaction.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/github-actions/raw/master/netcoreaction.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">This will open up the following YAML configuration file that we can edit to configure it according to the needs of our project:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/github-actions/raw/master/workflow.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/github-actions/raw/master/workflow.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/github-actions/blob/master/README.md#configuring-your-workflow-file" id="user-content-configuring-your-workflow-file" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>Configuring your workflow file</h2><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">The file below specifies building my .net core application using a specific folder and then I publish the binaries that will be available in the action's artifacts section. The package has been configured for Windows x64, .net core 3.1.6 and the package is self-contained (meaning that the .net core libraries will become part of the artifact).</p><div class="highlight highlight-source-yaml" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">my Project</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">on</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">push</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">branches</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">[ master ]</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">pull_request</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">branches</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">[ master ]</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">defaults</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">run</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">working-directory</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">myASPNETCoreApp</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">jobs</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">build</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">runs-on</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">ubuntu-latest</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">steps</span>:
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">uses</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">actions/checkout@v2</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Setup .NET Core</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">uses</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">actions/setup-dotnet@v1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">with</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">dotnet-version</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">3.1.302</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Install dependencies</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">run</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">dotnet restore</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Build Project</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">run</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">dotnet build --configuration Release --no-restore</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Test Project</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">run</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">dotnet test --no-restore --verbosity normal</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Publish</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">run</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">dotnet publish --configuration Debug --framework netcoreapp3.1 --runtime win-x64 --self-contained true</span>
- <span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">Upload a Build Artifact</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">uses</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">actions/upload-artifact@v2.1.1</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">with</span>:
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">name</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">myproject-win-64</span>
<span class="pl-ent" style="box-sizing: border-box; color: #22863a;">path</span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;">/home/runner/work/myproject/myproject/myASPNETCoreApp/bin/Debug/netcoreapp3.1/win-x64/publish/</span></pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Once completed, you will see the actions taking place on every commit. Below is a sample of execution for different changes in my project. Note that in the image below, the artifacts (binaries) of my project are attached to the output of the action's workflow.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/github-actions/raw/master/actionresults.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/github-actions/raw/master/actionresults.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">We can click on the commit and explore the output of the build and drill into the build log for more details:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/github-actions/raw/master/projectartifactis.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/github-actions/raw/master/projectartifactis.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">If we drill into the build log, we will be able to find the location of the machine where the artifacts are generated so we can zip them up. I've highlighted it in red and it should match the path mentioned in my YAML file.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/github-actions/raw/master/publishprocedure.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/github-actions/raw/master/publishprocedure.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">One thing to take into consideration is that this is not for free and there is a cost implied if you go over the limit. This build happens on a Linux machine (as defined in the YAML file) and there is a cost for using this machine. There is a very healthy limit and you can specify the amount of money you want to spend. The default is $0, so there is no risk of getting billed accidentally. More info <a href="https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">here</a>.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 0px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/github-actions/raw/master/aboutbilling.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/github-actions/raw/master/aboutbilling.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 0px; margin-top: 0px;">Source code can be find here:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 0px; margin-top: 0px;"><br /></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 0px; margin-top: 0px;">- <a href="https://github.com/JordiCorbilla/github-actions" style="background-color: transparent;">https://github.com/JordiCorbilla/github-actions</a></p>Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-60021693435544063942020-07-11T18:13:00.005+02:002024-02-11T19:38:01.875+01:00Predicting stock prices using a TensorFlow LSTM (long short-term memory) neural network for times series forecasting<h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLvjpR5WXkShNQOenH0cYesNO9Ihf4IjJ9PN40IBtPsdVjb6OKK883D2Jg_EfGkhqXQo_X-v5wKbWtbGfj9mk-Gb5P3btrLVDAydSIXCzljk-HlW4ZkrLLQ_1RaMWQDVUxxHcWhqWHDa9LQ0XfkxA5zzK_jq-UT64RtKaNyGRpgQ5S7EPFgcKuJse2Zrk/s1024/DALL%C2%B7E%202024-02-11%2018.37.05%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_Predicting%20Stock%20Prices%20Using%20a%20TensorFlow%20LSTM%20(Long%20Short-Term%20Memory)%20Neural%20Network%20for%20Time%20Seri.webp" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1024" data-original-width="1024" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLvjpR5WXkShNQOenH0cYesNO9Ihf4IjJ9PN40IBtPsdVjb6OKK883D2Jg_EfGkhqXQo_X-v5wKbWtbGfj9mk-Gb5P3btrLVDAydSIXCzljk-HlW4ZkrLLQ_1RaMWQDVUxxHcWhqWHDa9LQ0XfkxA5zzK_jq-UT64RtKaNyGRpgQ5S7EPFgcKuJse2Zrk/w200-h200/DALL%C2%B7E%202024-02-11%2018.37.05%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_Predicting%20Stock%20Prices%20Using%20a%20TensorFlow%20LSTM%20(Long%20Short-Term%20Memory)%20Neural%20Network%20for%20Time%20Seri.webp" width="200" /></a></div>1) Introduction</h2><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px; text-align: justify;">Predicting stock prices is a cumbersome task as it does not follow any specific pattern. Changes in the stock prices are purely based on supply and demand during a period of time. In order to learn the specific characteristics of a stock price, we can use deep learning to identify these patterns through machine learning. One of the most well-known networks for series forecasting is <a href="https://www.researchgate.net/publication/13853244_Long_Short-term_Memory" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">LSTM</a> (long short-term memory) which is a Recurrent Neural Network (RNN) that is able to remember information over a long period of time, thus making them extremely useful for predicting stock prices. RNNs are well-suited to time series data and they are able to process the data step-by-step, maintaining an internal state where they cache the information they have seen so far in a summarised version. The successful prediction of a stock's future price could yield a significant profit.</p><h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">2) Stock Market Data</h2><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">The initial data we will use for this model is taken directly from the <a href="https://finance.yahoo.com/quote/GOOG?p=GOOG" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">Yahoo Finance</a> page which contains the latest market data on a specific stock price. To perform this operation easily using Python, we will use the <a href="https://aroussi.com/post/python-yahoo-finance" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">yFinance</a> library which has been built specifically for this and that it will allow us to download all the information we need on a given <a href="https://www.investopedia.com/terms/t/tickersymbol.asp" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">ticker symbol</a>.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Below is a sample screenshot of the ticker symbol (GOOG) that we will use in this stock prediction article:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPZEJRIWRx_EbitZ_NF8RNtZ7b8mKPgFDWnpTxLkunK1A0MXbRwKkOkubw3AnS1nWte6hz6aALXyPicBvLomWBUBPWfVwC0FmE1pKF7causM8LeFLN_p1jhjH-2xzwtX37eoIVwEHcEOg/s1131/samplestock.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="682" data-original-width="1131" height="376" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPZEJRIWRx_EbitZ_NF8RNtZ7b8mKPgFDWnpTxLkunK1A0MXbRwKkOkubw3AnS1nWte6hz6aALXyPicBvLomWBUBPWfVwC0FmE1pKF7causM8LeFLN_p1jhjH-2xzwtX37eoIVwEHcEOg/w625-h376/samplestock.png" width="625" /></a></div><h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">2.1) Market Info Download</h3><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">To download the data info, we will need the yFinance library installed and then we will only need to perform the following operation to download all the relevant information of a given Stock using its ticker symbol.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Below is the output from the [download_market_data_info.py] file that is able to download financial data from Yahoo Finance.</p><div class="highlight highlight-source-batchfile" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">C:\Users\thund\Source\Repos\stock-prediction-deep-neural-learning<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">></span>python download_market_data_info.py
Info
{
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>52WeekChange<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 0.26037383,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>SandP52WeekChange<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 0.034871936,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>address1<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>1600 Amphitheatre Parkway<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>algorithm<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>annualHoldingsTurnover<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>annualReportExpenseRatio<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ask<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1432.77,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>askSize<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1400,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>averageDailyVolume10Day<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 2011171,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>averageVolume<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1857809,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>averageVolume10days<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 2011171,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>beta<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1.068946,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>beta3Year<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>bid<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1432.16,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>bidSize<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 3000,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>bookValue<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 297.759,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>category<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>circulatingSupply<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>city<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Mountain View<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>companyOfficers<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: [],
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>country<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>United States<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>currency<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>USD<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>dateShortInterest<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1592179200,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>dayHigh<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1441.19,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>dayLow<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1409.82,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>dividendRate<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>dividendYield<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>earningsQuarterlyGrowth<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 0.027,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>enterpriseToEbitda<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 17.899,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>enterpriseToRevenue<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 5.187,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>enterpriseValue<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 864533741568,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>exDividendDate<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>exchange<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>NMS<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>exchangeTimezoneName<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>America/New_York<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>exchangeTimezoneShortName<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>EDT<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>expireDate<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>fiftyDayAverage<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1417.009,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>fiftyTwoWeekHigh<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1532.106,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>fiftyTwoWeekLow<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1013.536,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>fiveYearAverageReturn<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>fiveYearAvgDividendYield<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>floatShares<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 613293304,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>forwardEps<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 55.05,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>forwardPE<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 26.028149,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>fromCurrency<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>fullTimeEmployees<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 123048,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>fundFamily<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>fundInceptionDate<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>gmtOffSetMilliseconds<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>-14400000<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>heldPercentInsiders<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 0.05746,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>heldPercentInstitutions<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 0.7062,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>industry<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Internet Content & Information<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>isEsgPopulated<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: false,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>lastCapGain<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>lastDividendValue<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>lastFiscalYearEnd<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1577750400,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>lastMarket<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>lastSplitDate<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1430092800,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>lastSplitFactor<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>10000000:10000000<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>legalType<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>logo_url<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>https://logo.clearbit.com/abc.xyz<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>longBusinessSummary<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Alphabet Inc. provides online advertising services in the United States, Europe, the Middle East, Africa, the Asia-Pacific, Canada, and Latin America. It offers performance and brand advertising services. The company operates through Google and Other Bets segments. The Google segment offers products, such as Ads, Android, Chrome, Google Cloud, Google Maps, Google Play, Hardware, Search, and YouTube, as well as technical infrastructure. It also offers digital content, cloud services, hardware devices, and other miscellaneous products and services. The Other Bets segment includes businesses, including Access, Calico, CapitalG, GV, Verily, Waymo, and X, as well as Internet and television services. Alphabet Inc. was founded in 1998 and is headquartered in Mountain View, California.<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>longName<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Alphabet Inc.<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>market<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>us_market<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>marketCap<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 979650805760,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>maxAge<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>maxSupply<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>messageBoardId<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>finmb_29096<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>morningStarOverallRating<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>morningStarRiskRating<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>mostRecentQuarter<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1585612800,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>navPrice<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>netIncomeToCommon<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 34522001408,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>nextFiscalYearEnd<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1640908800,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>open<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1411.1,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>openInterest<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>payoutRatio<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 0,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>pegRatio<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 4.38,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>phone<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>650-253-0000<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>previousClose<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1413.61,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>priceHint<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 2,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>priceToBook<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 4.812112,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>priceToSalesTrailing12Months<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 5.87754,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>profitMargins<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 0.20712,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>quoteType<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>EQUITY<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>regularMarketDayHigh<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1441.19,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>regularMarketDayLow<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1409.82,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>regularMarketOpen<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1411.1,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>regularMarketPreviousClose<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1413.61,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>regularMarketPrice<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1411.1,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>regularMarketVolume<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1084440,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>revenueQuarterlyGrowth<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>sector<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Communication Services<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>sharesOutstanding<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 336161984,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>sharesPercentSharesOut<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 0.0049,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>sharesShort<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 3371476,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>sharesShortPreviousMonthDate<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1589500800,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>sharesShortPriorMonth<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 3462105,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>shortName<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Alphabet Inc.<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>shortPercentOfFloat<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>shortRatio<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1.9,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>startDate<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>state<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>CA<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>strikePrice<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>symbol<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>GOOG<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>threeYearAverageReturn<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>toCurrency<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>totalAssets<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>tradeable<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: false,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>trailingAnnualDividendRate<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>trailingAnnualDividendYield<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>trailingEps<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 49.572,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>trailingPE<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 28.904415,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>twoHundredDayAverage<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1352.9939,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>volume<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: 1084440,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>volume24Hr<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>volumeAllCurrencies<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>website<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>http://www.abc.xyz<span class="pl-pds" style="box-sizing: border-box;">"</span></span>,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>yield<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>ytdReturn<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: null,
<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>zip<span class="pl-pds" style="box-sizing: border-box;">"</span></span>: <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>94043<span class="pl-pds" style="box-sizing: border-box;">"</span></span>
}
ISIN
-
Major Holders
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span> 5.75<span class="pl-smi" style="box-sizing: border-box;">% %</span> of Shares Held by All Insider
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span> 70.62<span class="pl-smi" style="box-sizing: border-box;">% %</span> of Shares Held by Institutions
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2</span> 74.93<span class="pl-smi" style="box-sizing: border-box;">% %</span> of Float Held by Institutions
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3304</span> Number of Institutions Holding Shares
Institutional Holders
Holder Shares <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">Date</span> Reported % Out Value
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span> Vanguard Group, Inc. (The) <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">23162950</span> 2020-03-30 0.0687 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">26934109889</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span> Blackrock Inc. <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">20264225</span> 2020-03-30 0.0601 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">23563443472</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2</span> Price (T.Rowe) Associates Inc <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">12520058</span> 2020-03-30 0.0371 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">14558448642</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3</span> State Street Corporation <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">11814026</span> 2020-03-30 0.0350 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">13737467573</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4</span> FMR, LLC <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">8331868</span> 2020-03-30 0.0247 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">9688379429</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5</span> Capital International Investors <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4555880</span> 2020-03-30 0.0135 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5297622822</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">6</span> Geode Capital Management, LLC <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4403934</span> 2020-03-30 0.0131 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5120938494</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">7</span> Northern Trust Corporation <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4017009</span> 2020-03-30 0.0119 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4671018235</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">8</span> JP Morgan Chase <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">&</span> Company <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3707376</span> 2020-03-30 0.0110 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4310973886</span>
<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">9</span> AllianceBernstein, L.P. <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3483382</span> 2020-03-30 0.0103 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4050511423</span>
Dividents
Series([], Name: Dividends, dtype: int64)
Splits
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">Date</span>
2014-03-27 2.002
2015-04-27 1.000
Name: Stock Splits, dtype: float64
Actions
Dividends Stock Splits
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">Date</span>
2014-03-27 0.0 2.002
2015-04-27 0.0 1.000
Calendar
Empty DataFrame
Columns: []
Index: [Earnings Date, Earnings Average, Earnings Low, Earnings High, Revenue Average, Revenue Low, Revenue High]
Recommendations
Firm To Grade From Grade Action
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">Date</span>
2012-03-14 15:28:00 Oxen Group Hold init
2012-03-28 06:29:00 Citigroup Buy main
2012-04-03 08:45:00 Global Equities Research Overweight main
2012-04-05 06:34:00 Deutsche Bank Buy main
2012-04-09 06:03:00 Pivotal Research Buy main
2012-04-10 11:32:00 UBS Buy main
2012-04-13 06:16:00 Deutsche Bank Buy main
2012-04-13 06:18:00 Jefferies Buy main
2012-04-13 06:37:00 PiperJaffray Overweight main
2012-04-13 06:38:00 Goldman Sachs Neutral main
2012-04-13 06:41:00 JP Morgan Overweight main
2012-04-13 06:51:00 Oppenheimer Outperform main
2012-04-13 07:13:00 Benchmark Hold main
2012-04-13 08:46:00 BMO Capital Outperform main
2012-04-16 06:52:00 Hilliard Lyons Buy main
2012-06-06 06:17:00 Deutsche Bank Buy main
2012-06-06 06:56:00 JP Morgan Overweight main
2012-06-22 06:15:00 Citigroup Buy main
2012-07-13 05:57:00 Wedbush Neutral init
2012-07-17 09:33:00 Outperform main
2012-07-20 06:43:00 Benchmark Hold main
2012-07-20 06:54:00 Deutsche Bank Buy main
2012-07-20 06:59:00 Bank of America Buy main
2012-08-13 05:49:00 Morgan Stanley Overweight Equal-Weight up
2012-09-17 06:07:00 Global Equities Research Overweight main
2012-09-21 06:28:00 Cantor Fitzgerald Buy init
2012-09-24 06:11:00 Citigroup Buy main
2012-09-24 09:05:00 Pivotal Research Buy main
2012-09-25 07:20:00 Capstone Buy main
2012-09-26 05:48:00 Canaccord Genuity Buy main
... ... ... ... ...
2017-10-27 19:29:31 UBS Buy main
2018-02-02 14:04:52 PiperJaffray Overweight Overweight main
2018-04-24 11:43:49 JP Morgan Overweight Overweight main
2018-04-24 12:24:37 Deutsche Bank Buy Buy main
2018-05-05 14:00:37 B. Riley FBR Buy main
2018-07-13 13:49:13 Cowen <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">&</span> Co. Outperform Outperform main
2018-07-24 11:50:55 Cowen <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">&</span> Co. Outperform Outperform main
2018-07-24 13:33:47 Raymond James Outperform Outperform main
2018-10-23 11:18:00 Deutsche Bank Buy Buy main
2018-10-26 15:17:08 Raymond James Outperform Outperform main
2019-01-23 12:55:04 Deutsche Bank Buy Buy main
2019-02-05 12:55:12 Deutsche Bank Buy Buy main
2019-02-05 13:18:47 PiperJaffray Overweight Overweight main
2019-05-15 12:34:54 Deutsche Bank Buy main
2019-10-23 12:58:59 Credit Suisse Outperform main
2019-10-29 11:58:09 Raymond James Outperform main
2019-10-29 14:15:40 Deutsche Bank Buy main
2019-10-29 15:48:29 UBS Buy main
2020-01-06 11:22:07 Pivotal Research Buy Hold up
2020-01-17 13:01:48 UBS Buy main
2020-02-04 12:26:56 Piper Sandler Overweight main
2020-02-04 12:41:00 Raymond James Outperform main
2020-02-04 14:00:36 Deutsche Bank Buy main
2020-02-06 11:34:20 CFRA Strong Buy main
2020-03-18 13:52:51 JP Morgan Overweight main
2020-03-30 13:26:16 UBS Buy main
2020-04-17 13:01:41 Oppenheimer Outperform main
2020-04-20 19:29:50 Credit Suisse Outperform main
2020-04-29 14:01:51 UBS Buy main
2020-05-05 12:44:16 Deutsche Bank Buy main
[219 rows x <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4</span> columns]
Earnings
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []
Quarterly Earnings
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []
Financials
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []
Quarterly Financials
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []
Balance Sheet
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []
Quarterly Balance Sheet
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []
Balancesheet
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []
Quarterly Balancesheet
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []
Cashflow
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []
Quarterly Cashflow
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []
Sustainability
None
Options
('2020-07-02', '2020-07-10', '2020-07-17', '2020-07-24', '2020-07-31', '2020-08-07', '2020-08-21', '2020-09-18', '2020-11-20', '2020-12-01', '2020-12-18', '2021-01-15', '2021-06-18', '2022-01-21', '2022-06-17')</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">The data has a JSON document that we could use later on to create our Security Master if we ever wanted to store this data somewhere to keep track of the Securities we are going to trade with. As the data could come with different fields, my suggestion is to store them on a Data Lake so we can build it from multiple sources without having to worry too much about the way the data is structured.</p><h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning#22-market-data-download" id="user-content-22-market-data-download" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>2.2) Market Data Download</h3><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">The previous step helps us to identify several characteristics of a given ticker symbol so we can use its properties to define some of the charts I'm showing below. Note that the yFinance library only requires the stock to download via ticker symbol, the start date and end date of the period we want to get. Additionally, we can also specify the granularity of the data using the interval parameter. By default, the interval is 1 day and this is the one I will use for my training.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">To download the data we can use the following command:</p><div class="highlight highlight-source-python" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-s1" style="box-sizing: border-box;">start</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> <span class="pl-s1" style="box-sizing: border-box;">pd</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">to_datetime</span>(<span class="pl-s" style="box-sizing: border-box; color: #032f62;">'2004-08-01'</span>)
<span class="pl-s1" style="box-sizing: border-box;">stock</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> [<span class="pl-s" style="box-sizing: border-box; color: #032f62;">'GOOG'</span>]
<span class="pl-s1" style="box-sizing: border-box;">data</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> <span class="pl-s1" style="box-sizing: border-box;">yf</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">download</span>(<span class="pl-s1" style="box-sizing: border-box;">stock</span>, <span class="pl-s1" style="box-sizing: border-box;">start</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s1" style="box-sizing: border-box;">start</span>, <span class="pl-s1" style="box-sizing: border-box;">end</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s1" style="box-sizing: border-box;">datetime</span>.<span class="pl-s1" style="box-sizing: border-box;">date</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">today</span>())
<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">print</span>(<span class="pl-s1" style="box-sizing: border-box;">data</span>)</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">And the sample output:</p><div class="highlight highlight-source-batchfile" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">C:\Users\thund\Source\Repos\stock-prediction-deep-neural-learning<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">></span>python download_market_data.py
[*********************100%<span class="pl-v" style="box-sizing: border-box; color: #e36209;">*</span>**********************] <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span> of <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span> completed
Open High Low Close Adj Close Volume
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">Date</span>
2004-08-19 49.813286 51.835709 47.800831 49.982655 49.982655 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">44871300</span>
2004-08-20 50.316402 54.336334 50.062355 53.952770 53.952770 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">22942800</span>
2004-08-23 55.168217 56.528118 54.321388 54.495735 54.495735 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">18342800</span>
2004-08-24 55.412300 55.591629 51.591621 52.239193 52.239193 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">15319700</span>
2004-08-25 52.284027 53.798351 51.746044 52.802086 52.802086 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">9232100</span>
2004-08-26 52.279045 53.773445 52.134586 53.753517 53.753517 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">7128600</span>
2004-08-27 53.848164 54.107193 52.647663 52.876804 52.876804 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">6241200</span>
2004-08-30 52.443428 52.548038 50.814533 50.814533 50.814533 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5221400</span>
2004-08-31 50.958992 51.661362 50.889256 50.993862 50.993862 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4941200</span>
2004-09-01 51.158245 51.292744 49.648903 49.937820 49.937820 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">9181600</span>
2004-09-02 49.409801 50.993862 49.285267 50.565468 50.565468 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">15190400</span>
2004-09-03 50.286514 50.680038 49.474556 49.818268 49.818268 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5176800</span>
2004-09-07 50.316402 50.809555 49.619015 50.600338 50.600338 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5875200</span>
2004-09-08 50.181908 51.322632 50.062355 50.958992 50.958992 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">5009200</span>
2004-09-09 51.073563 51.163227 50.311420 50.963974 50.963974 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4080900</span>
2004-09-10 50.610302 53.081039 50.460861 52.468334 52.468334 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">8740200</span>
2004-09-13 53.115910 54.002586 53.031227 53.549286 53.549286 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">7881300</span>
2004-09-14 53.524376 55.790882 53.195610 55.536835 55.536835 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">10880300</span>
2004-09-15 55.073570 56.901718 54.894241 55.790882 55.790882 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">10763900</span>
2004-09-16 55.960247 57.683788 55.616535 56.772205 56.772205 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">9310200</span>
2004-09-17 56.996365 58.525631 56.562988 58.525631 58.525631 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">9517400</span>
2004-09-20 58.256641 60.572956 58.166977 59.457142 59.457142 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">10679200</span>
2004-09-21 59.681301 59.985161 58.535595 58.699978 58.699978 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">7263000</span>
2004-09-22 58.480801 59.611561 58.186901 58.968971 58.968971 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">7617100</span>
2004-09-23 59.198112 61.086033 58.291508 60.184414 60.184414 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">8576100</span>
2004-09-24 60.244190 61.818291 59.656395 59.691261 59.691261 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">9166700</span>
2004-09-27 59.556767 60.214302 58.680054 58.909195 58.909195 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">7099600</span>
2004-09-28 60.423519 63.462128 59.880554 63.193138 63.193138 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">17009400</span>
2004-09-29 63.113434 67.257904 62.879314 65.295258 65.295258 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">30661400</span>
2004-09-30 64.707458 65.902977 64.259140 64.558022 64.558022 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">13823300</span>
... ... ... ... ... ... ...
2020-05-19 1386.996948 1392.000000 1373.484985 1373.484985 1373.484985 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1280600</span>
2020-05-20 1389.579956 1410.420044 1387.250000 1406.719971 1406.719971 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1655400</span>
2020-05-21 1408.000000 1415.489990 1393.449951 1402.800049 1402.800049 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1385000</span>
2020-05-22 1396.709961 1412.760010 1391.829956 1410.420044 1410.420044 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1309400</span>
2020-05-26 1437.270020 1441.000000 1412.130005 1417.020020 1417.020020 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2060600</span>
2020-05-27 1417.250000 1421.739990 1391.290039 1417.839966 1417.839966 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1685800</span>
2020-05-28 1396.859985 1440.839966 1396.000000 1416.729980 1416.729980 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1692200</span>
2020-05-29 1416.939941 1432.569946 1413.349976 1428.920044 1428.920044 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1838100</span>
2020-06-01 1418.390015 1437.959961 1418.000000 1431.819946 1431.819946 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1217100</span>
2020-06-02 1430.550049 1439.609985 1418.829956 1439.219971 1439.219971 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1278100</span>
2020-06-03 1438.300049 1446.552002 1429.776978 1436.380005 1436.380005 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1256200</span>
2020-06-04 1430.400024 1438.959961 1404.729980 1412.180054 1412.180054 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1484300</span>
2020-06-05 1413.170044 1445.050049 1406.000000 1438.390015 1438.390015 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1734900</span>
2020-06-08 1422.339966 1447.989990 1422.339966 1446.609985 1446.609985 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1404200</span>
2020-06-09 1445.359985 1468.000000 1443.209961 1456.160034 1456.160034 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1409200</span>
2020-06-10 1459.540039 1474.259033 1456.270020 1465.849976 1465.849976 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1525200</span>
2020-06-11 1442.479980 1454.474976 1402.000000 1403.839966 1403.839966 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1991300</span>
2020-06-12 1428.489990 1437.000000 1386.020020 1413.180054 1413.180054 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1944200</span>
2020-06-15 1390.800049 1424.800049 1387.920044 1419.849976 1419.849976 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1503900</span>
2020-06-16 1445.219971 1455.020020 1425.900024 1442.719971 1442.719971 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1709200</span>
2020-06-17 1447.160034 1460.000000 1431.380005 1451.119995 1451.119995 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1548300</span>
2020-06-18 1449.160034 1451.410034 1427.010010 1435.959961 1435.959961 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1581900</span>
2020-06-19 1444.000000 1447.800049 1421.349976 1431.719971 1431.719971 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3157900</span>
2020-06-22 1429.000000 1452.750000 1423.209961 1451.859985 1451.859985 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1542400</span>
2020-06-23 1455.640015 1475.941040 1445.239990 1464.410034 1464.410034 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1429800</span>
2020-06-24 1461.510010 1475.420044 1429.750000 1431.969971 1431.969971 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1756000</span>
2020-06-25 1429.900024 1442.900024 1420.000000 1441.329956 1441.329956 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1230500</span>
2020-06-26 1431.390015 1433.449951 1351.989990 1359.900024 1359.900024 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">4267700</span>
2020-06-29 1358.180054 1395.599976 1347.010010 1394.969971 1394.969971 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1810200</span>
2020-06-30 1390.439941 1418.650024 1383.959961 1413.609985 1413.609985 <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">2041600</span>
[3994 rows x <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">6</span> columns]</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Note that is important to mention the start date correctly just to ensure we are collecting data. If we don't do that we might end up having some NaN variables that could affect the output of our training.</p><h2 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">3) Deep Learning Model</h2><h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning#31-training-and-validation-data" id="user-content-31-training-and-validation-data" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>3.1) Training and Validation Data</h3><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px; text-align: justify;">Now that we have the data that we want to use, we need to define what defines our training and validation data. As stocks could vary depending on the dates, the function I have created requires 3 basic arguments:</p><ul style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px; padding-left: 2em;"><li style="box-sizing: border-box;">Ticker Symbol: <span style="box-sizing: border-box; font-weight: 600;">GOOG</span></li><li style="box-sizing: border-box; margin-top: 0.25em;">Start Date: Date as to when they started, in this case, it was <span style="box-sizing: border-box; font-weight: 600;">2004-Aug-01</span>.</li><li style="box-sizing: border-box; margin-top: 0.25em;">Validation Date: Date as to when we want the validation to be considered. In this case, we specify <span style="box-sizing: border-box; font-weight: 600;">2017-01-01</span> as our data point.</li></ul><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Note that you will need to have configured <a href="https://www.tensorflow.org/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">TensorFlow</a>, Keras, and a GPU in order to run the samples below.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">In this exercise, I'm only interested in the <a href="https://www.investopedia.com/terms/c/closingprice.asp" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">closing price</a> which is the standard benchmark regarding stocks or securities.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Below you can find the chart with the division we will create between Training Data and Validation Data:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8rEZehgxaJbBOOYP2aVMwgzuasYVK1fBZn2A1RzbAuT1qXHnXkeAo75p1_50j5X6BW_7MTnJKkHpb2zSsQF9RyxC14fr3wqO8D3cKZnmZ6eCCF0-bN-PNSaLZs9HFR_lFX8GY9Mo7cE4/s1200/TrainingDataValidationData.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="500" data-original-width="1200" height="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8rEZehgxaJbBOOYP2aVMwgzuasYVK1fBZn2A1RzbAuT1qXHnXkeAo75p1_50j5X6BW_7MTnJKkHpb2zSsQF9RyxC14fr3wqO8D3cKZnmZ6eCCF0-bN-PNSaLZs9HFR_lFX8GY9Mo7cE4/w625-h260/TrainingDataValidationData.png" width="625" /></a></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Also, the histogram showing the distribution of the prices:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhR0cZJ0X9-D3fPZjo7mgEGdgZsuUAWmN5DCoq8JMiTkguwnoXA1Zlpzp_nrFlAC3CIR-ylecJ6REyn1N0u-5iDnFfV7_ggHx1sgqD4PygqkeBha8wJ7VUII7ozculqvbk8XWPryfKGPzI/s640/DataHistogram.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="640" height="375" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhR0cZJ0X9-D3fPZjo7mgEGdgZsuUAWmN5DCoq8JMiTkguwnoXA1Zlpzp_nrFlAC3CIR-ylecJ6REyn1N0u-5iDnFfV7_ggHx1sgqD4PygqkeBha8wJ7VUII7ozculqvbk8XWPryfKGPzI/w500-h375/DataHistogram.png" width="500" /></a></div><div><h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">3.2) Data Normalization</h3><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px; text-align: justify;">In order to normalize the data, we need to scale it between 0 and 1 so we talk on a common scale. To accomplish this, we can use the preprocessing tool <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">MinMaxScaler</a> as seen below:</p><div class="highlight highlight-source-python" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"> <span class="pl-s1" style="box-sizing: border-box;">min_max</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> <span class="pl-v" style="box-sizing: border-box; color: #e36209;">MinMaxScaler</span>(<span class="pl-s1" style="box-sizing: border-box;">feature_range</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span>(<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span>, <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>))
<span class="pl-s1" style="box-sizing: border-box;">train_scaled</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> <span class="pl-s1" style="box-sizing: border-box;">min_max</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">fit_transform</span>(<span class="pl-s1" style="box-sizing: border-box;">training_data</span>)</pre></div><h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning#33-adding-timesteps" id="user-content-33-adding-timesteps" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>3.3) Adding Timesteps</h3><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px; text-align: justify;">LSTM network needs the data imported as a 3D array. To translate this 2D array into a 3D one, we use a short <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;">timestep</a> to loop through the data and create smaller partitions and feed them into the model. The final array is then reshaped into training samples, x number of timesteps, and 1 feature per step. The code below represents this concept:</p><div class="highlight highlight-source-python" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"> <span class="pl-s1" style="box-sizing: border-box;">time_steps</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3</span>
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">for</span> <span class="pl-s1" style="box-sizing: border-box;">i</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">in</span> <span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">range</span>(<span class="pl-s1" style="box-sizing: border-box;">time_steps</span>, <span class="pl-s1" style="box-sizing: border-box;">train_scaled</span>.<span class="pl-s1" style="box-sizing: border-box;">shape</span>[<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span>]):
<span class="pl-s1" style="box-sizing: border-box;">x_train</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">append</span>(<span class="pl-s1" style="box-sizing: border-box;">train_scaled</span>[<span class="pl-s1" style="box-sizing: border-box;">i</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">-</span> <span class="pl-s1" style="box-sizing: border-box;">time_steps</span>:<span class="pl-s1" style="box-sizing: border-box;">i</span>])
<span class="pl-s1" style="box-sizing: border-box;">y_train</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">append</span>(<span class="pl-s1" style="box-sizing: border-box;">train_scaled</span>[<span class="pl-s1" style="box-sizing: border-box;">i</span>, <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span>])</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/blob/master/3dmatrix.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/raw/master/3dmatrix.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></p></div><div><h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;"><p style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; font-weight: 400; margin-bottom: 16px; margin-top: 0px; text-align: justify;">We have implemented a time step of 3 days. Using this technique, we allow our network to look back 3 days on our data to predict the subsequent day). The figure below represents how our implementation uses this concept and how the first 3 samples for Close price would generate the 4th sample and so on. This will generate a matrix of shape (3,1), 3 being the time steps, and 1 the number of features (Close price).</p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/blob/master/timesteps.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; margin-left: 1em; margin-right: 1em; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/raw/master/timesteps.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></div><p style="box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 16px; font-weight: 400; margin-bottom: 16px; margin-top: 0px;"></p></h3><h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;">3.4) Creation of the deep learning model LSTM</h3><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">To create this model, you will need to have <span style="box-sizing: border-box; font-weight: 600;">TensorFlow</span>, <span style="box-sizing: border-box; font-weight: 600;">TensorFlow-Gpu</span> and <span style="box-sizing: border-box; font-weight: 600;">Keras</span> installed in order for this to run. The code for this model can be seen below and the explanation for each layer is also defined below:</p><div class="highlight highlight-source-python" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">def</span> <span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">create_long_short_term_memory_model</span>(<span class="pl-s1" style="box-sizing: border-box;">x_train</span>):
<span class="pl-s1" style="box-sizing: border-box;">model</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> <span class="pl-v" style="box-sizing: border-box; color: #e36209;">Sequential</span>()
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># 1st layer with Dropout regularisation</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># * units = add 100 neurons is the dimensionality of the output space</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># * return_sequences = True to stack LSTM layers so the next LSTM layer has a three-dimensional sequence input</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># * input_shape => Shape of the training dataset</span>
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">add</span>(<span class="pl-v" style="box-sizing: border-box; color: #e36209;">LSTM</span>(<span class="pl-s1" style="box-sizing: border-box;">units</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">100</span>, <span class="pl-s1" style="box-sizing: border-box;">return_sequences</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">True</span>, <span class="pl-s1" style="box-sizing: border-box;">input_shape</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span>(<span class="pl-s1" style="box-sizing: border-box;">x_train</span>.<span class="pl-s1" style="box-sizing: border-box;">shape</span>[<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>], <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>)))
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># 20% of the layers will be dropped</span>
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">add</span>(<span class="pl-v" style="box-sizing: border-box; color: #e36209;">Dropout</span>(<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0.2</span>))
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># 2nd LSTM layer</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># * units = add 50 neurons is the dimensionality of the output space</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># * return_sequences = True to stack LSTM layers so the next LSTM layer has a three-dimensional sequence input</span>
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">add</span>(<span class="pl-v" style="box-sizing: border-box; color: #e36209;">LSTM</span>(<span class="pl-s1" style="box-sizing: border-box;">units</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">50</span>, <span class="pl-s1" style="box-sizing: border-box;">return_sequences</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">True</span>))
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># 20% of the layers will be dropped</span>
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">add</span>(<span class="pl-v" style="box-sizing: border-box; color: #e36209;">Dropout</span>(<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0.2</span>))
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># 3rd LSTM layer</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># * units = add 50 neurons is the dimensionality of the output space</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># * return_sequences = True to stack LSTM layers so the next LSTM layer has a three-dimensional sequence input</span>
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">add</span>(<span class="pl-v" style="box-sizing: border-box; color: #e36209;">LSTM</span>(<span class="pl-s1" style="box-sizing: border-box;">units</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">50</span>, <span class="pl-s1" style="box-sizing: border-box;">return_sequences</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">True</span>))
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># 50% of the layers will be dropped</span>
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">add</span>(<span class="pl-v" style="box-sizing: border-box; color: #e36209;">Dropout</span>(<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0.5</span>))
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># 4th LSTM layer</span>
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># * units = add 50 neurons is the dimensionality of the output space</span>
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">add</span>(<span class="pl-v" style="box-sizing: border-box; color: #e36209;">LSTM</span>(<span class="pl-s1" style="box-sizing: border-box;">units</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">50</span>))
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># 50% of the layers will be dropped</span>
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">add</span>(<span class="pl-v" style="box-sizing: border-box; color: #e36209;">Dropout</span>(<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0.5</span>))
<span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># Dense layer that specifies an output of one unit</span>
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">add</span>(<span class="pl-v" style="box-sizing: border-box; color: #e36209;">Dense</span>(<span class="pl-s1" style="box-sizing: border-box;">units</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>))
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">summary</span>()
<span class="pl-s1" style="box-sizing: border-box;">tf</span>.<span class="pl-s1" style="box-sizing: border-box;">keras</span>.<span class="pl-s1" style="box-sizing: border-box;">utils</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">plot_model</span>(<span class="pl-s1" style="box-sizing: border-box;">model</span>, <span class="pl-s1" style="box-sizing: border-box;">to_file</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s1" style="box-sizing: border-box;">os</span>.<span class="pl-s1" style="box-sizing: border-box;">path</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">join</span>(<span class="pl-s1" style="box-sizing: border-box;">project_folder</span>, <span class="pl-s" style="box-sizing: border-box; color: #032f62;">'model_lstm.png'</span>), <span class="pl-s1" style="box-sizing: border-box;">show_shapes</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">True</span>,
<span class="pl-s1" style="box-sizing: border-box;">show_layer_names</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">True</span>)
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">return</span> <span class="pl-s1" style="box-sizing: border-box;">model</span></pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">The rendered model can be seen in the image below, producing a model with more than 100k trainable parameters. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/raw/master/model_lstm.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; margin-left: 1em; margin-right: 1em; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/raw/master/model_lstm.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"></p><div class="highlight highlight-source-batchfile" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">Layer (type) Output Shape Param #
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">================================================================</span>=
lstm_1 (LSTM) (None, 60, 100) <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">40800</span>
_________________________________________________________________
dropout_1 (Dropout) (None, 60, 100) <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span>
_________________________________________________________________
lstm_2 (LSTM) (None, 60, 50) <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">30200</span>
_________________________________________________________________
dropout_2 (Dropout) (None, 60, 50) <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span>
_________________________________________________________________
lstm_3 (LSTM) (None, 60, 50) <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">20200</span>
_________________________________________________________________
dropout_3 (Dropout) (None, 60, 50) <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span>
_________________________________________________________________
lstm_4 (LSTM) (None, 50) <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">20200</span>
_________________________________________________________________
dropout_4 (Dropout) (None, 50) <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span>
_________________________________________________________________
dense_1 (Dense) (None, 1) <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">51</span>
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">================================================================</span>=
Total params: 111,451
Trainable params: 111,451
Non-trainable params: <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span></pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Once we have defined the model, we need to specify the metrics we want to use to track how well our model is behaving and also the kind of optimizer we want to use for our training. I have also defined the patience I want my model to have and what is the rule defined for it.</p><div class="highlight highlight-source-python" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"> <span class="pl-s1" style="box-sizing: border-box;">defined_metrics</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> [
<span class="pl-s1" style="box-sizing: border-box;">tf</span>.<span class="pl-s1" style="box-sizing: border-box;">keras</span>.<span class="pl-s1" style="box-sizing: border-box;">metrics</span>.<span class="pl-v" style="box-sizing: border-box; color: #e36209;">MeanSquaredError</span>(<span class="pl-s1" style="box-sizing: border-box;">name</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s" style="box-sizing: border-box; color: #032f62;">'MSE'</span>)
]
<span class="pl-s1" style="box-sizing: border-box;">callback</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> <span class="pl-s1" style="box-sizing: border-box;">tf</span>.<span class="pl-s1" style="box-sizing: border-box;">keras</span>.<span class="pl-s1" style="box-sizing: border-box;">callbacks</span>.<span class="pl-v" style="box-sizing: border-box; color: #e36209;">EarlyStopping</span>(<span class="pl-s1" style="box-sizing: border-box;">monitor</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s" style="box-sizing: border-box; color: #032f62;">'val_loss'</span>, <span class="pl-s1" style="box-sizing: border-box;">patience</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3</span>, <span class="pl-s1" style="box-sizing: border-box;">mode</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s" style="box-sizing: border-box; color: #032f62;">'min'</span>, <span class="pl-s1" style="box-sizing: border-box;">verbose</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>)
<span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">compile</span>(<span class="pl-s1" style="box-sizing: border-box;">optimizer</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s" style="box-sizing: border-box; color: #032f62;">'adam'</span>, <span class="pl-s1" style="box-sizing: border-box;">loss</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s" style="box-sizing: border-box; color: #032f62;">'mean_squared_error'</span>, <span class="pl-s1" style="box-sizing: border-box;">metrics</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s1" style="box-sizing: border-box;">defined_metrics</span>)
<span class="pl-s1" style="box-sizing: border-box;">history</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> <span class="pl-s1" style="box-sizing: border-box;">model</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">fit</span>(<span class="pl-s1" style="box-sizing: border-box;">x_train</span>, <span class="pl-s1" style="box-sizing: border-box;">y_train</span>, <span class="pl-s1" style="box-sizing: border-box;">epochs</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s1" style="box-sizing: border-box;">epochs</span>, <span class="pl-s1" style="box-sizing: border-box;">batch_size</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span><span class="pl-s1" style="box-sizing: border-box;">batch_size</span>, <span class="pl-s1" style="box-sizing: border-box;">validation_data</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span>(<span class="pl-s1" style="box-sizing: border-box;">x_test</span>, <span class="pl-s1" style="box-sizing: border-box;">y_test</span>),
<span class="pl-s1" style="box-sizing: border-box;">callbacks</span><span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span>[<span class="pl-s1" style="box-sizing: border-box;">callback</span>])</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">This model is slightly fined tuned to reach the lowest validation loss. In this example, we reach a validation loss of 0.14% with an MSE (Mean Square Error) of 0.14% which is relatively good, providing us with a very accurate result.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">The training result can be seen below:</p><div class="highlight highlight-source-batchfile" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">Train on <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">3055</span> samples, validate on <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">881</span> samples
Epoch 1/100
2020-07-11 15:15:34.557035: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
3112/3112 [<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">==============================</span>] - 19s 6ms/sample - loss: 0.0451 - MSE: 0.0451 - val_loss: 0.0068 - val_MSE: 0.0068
Epoch 2/100
3112/3112 [<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">==============================</span>] - 4s 1ms/sample - loss: 0.0088 - MSE: 0.0088 - val_loss: 0.0045 - val_MSE: 0.0045
Epoch 3/100
3112/3112 [<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">==============================</span>] - 5s 1ms/sample - loss: 0.0062 - MSE: 0.0062 - val_loss: 0.0032 - val_MSE: 0.0032
Epoch 4/100
3112/3112 [<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">==============================</span>] - 5s 1ms/sample - loss: 0.0051 - MSE: 0.0051 - val_loss: 0.0015 - val_MSE: 0.0015
Epoch 5/100
3112/3112 [<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">==============================</span>] - 7s 2ms/sample - loss: 0.0045 - MSE: 0.0045 - val_loss: 0.0013 - val_MSE: 0.0013
Epoch 6/100
3112/3112 [<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">==============================</span>] - 5s 2ms/sample - loss: 0.0045 - MSE: 0.0045 - val_loss: 0.0013 - val_MSE: 0.0013
Epoch 7/100
3112/3112 [<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">==============================</span>] - 5s 2ms/sample - loss: 0.0045 - MSE: 0.0045 - val_loss: 0.0015 - val_MSE: 0.0015
Epoch 8/100
3112/3112 [<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">==============================</span>] - 5s 1ms/sample - loss: 0.0040 - MSE: 0.0040 - val_loss: 0.0015 - val_MSE: 0.0015
Epoch 9/100
3112/3112 [<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">==============================</span>] - 5s 1ms/sample - loss: 0.0039 - MSE: 0.0039 - val_loss: 0.0014 - val_MSE: 0.0014
Epoch 00009: early stopping
saving weights
plotting loss
plotting MSE
display the content of the model
886/1 - 0s - loss: 0.0029 - MSE: 0.0014
loss : 0.0014113364930413916
MSE : 0.0014113366</pre></div><h3 style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 1.25em; line-height: 1.25; margin-bottom: 16px; margin-top: 24px;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning#35-making-predictions-happen" id="user-content-35-making-predictions-happen" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>3.5) Making predictions happen</h3><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Now it is time to prepare our testing data and send it through our deep-learning model to obtain the predictions we are trying to get.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">First, we need to import the test data using the same approach we used for the training data using the time steps:</p><div class="highlight highlight-source-python" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"> <span class="pl-c" style="box-sizing: border-box; color: #6a737d;"># Testing Data Transformation</span>
<span class="pl-s1" style="box-sizing: border-box;">x_test</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> []
<span class="pl-s1" style="box-sizing: border-box;">y_test</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> []
<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">for</span> <span class="pl-s1" style="box-sizing: border-box;">i</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">in</span> <span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">range</span>(<span class="pl-s1" style="box-sizing: border-box;">time_steps</span>, <span class="pl-s1" style="box-sizing: border-box;">test_scaled</span>.<span class="pl-s1" style="box-sizing: border-box;">shape</span>[<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span>]):
<span class="pl-s1" style="box-sizing: border-box;">x_test</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">append</span>(<span class="pl-s1" style="box-sizing: border-box;">test_scaled</span>[<span class="pl-s1" style="box-sizing: border-box;">i</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">-</span> <span class="pl-s1" style="box-sizing: border-box;">time_steps</span>:<span class="pl-s1" style="box-sizing: border-box;">i</span>])
<span class="pl-s1" style="box-sizing: border-box;">y_test</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">append</span>(<span class="pl-s1" style="box-sizing: border-box;">test_scaled</span>[<span class="pl-s1" style="box-sizing: border-box;">i</span>, <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span>])
<span class="pl-s1" style="box-sizing: border-box;">x_test</span>, <span class="pl-s1" style="box-sizing: border-box;">y_test</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> <span class="pl-s1" style="box-sizing: border-box;">np</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">array</span>(<span class="pl-s1" style="box-sizing: border-box;">x_test</span>), <span class="pl-s1" style="box-sizing: border-box;">np</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">array</span>(<span class="pl-s1" style="box-sizing: border-box;">y_test</span>)
<span class="pl-s1" style="box-sizing: border-box;">x_test</span> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">=</span> <span class="pl-s1" style="box-sizing: border-box;">np</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">reshape</span>(<span class="pl-s1" style="box-sizing: border-box;">x_test</span>, (<span class="pl-s1" style="box-sizing: border-box;">x_test</span>.<span class="pl-s1" style="box-sizing: border-box;">shape</span>[<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">0</span>], <span class="pl-s1" style="box-sizing: border-box;">x_test</span>.<span class="pl-s1" style="box-sizing: border-box;">shape</span>[<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>], <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">1</span>))</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Now we can call the predict method which will allow us to generate the stock prediction based on the training done over the training data. As a result, we will generate a CSV file that contains the result of the prediction and also a chart that shows what's the real vs the estimation.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/raw/master/Alphabet%20Inc_prediction.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/raw/master/Alphabet%20Inc_prediction.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">With the validation loss and validation MSE metrics:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;"><a href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/raw/master/loss.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/raw/master/loss.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a> <a href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/raw/master/MSE.png" rel="noopener noreferrer" style="background-color: initial; box-sizing: border-box; color: #0366d6; text-decoration-line: none;" target="_blank"><img alt="" src="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/raw/master/MSE.png" style="border-style: none; box-sizing: initial; max-width: 100%;" /></a></p><h1 style="background-color: white; border-bottom: 1px solid rgb(234, 236, 239); box-sizing: border-box; color: #24292e; line-height: 1.25; margin: 24px 0px 16px; padding-bottom: 0.3em;"><a aria-hidden="true" class="anchor" href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning#4-usage" id="user-content-4-usage" style="background-color: initial; box-sizing: border-box; color: #0366d6; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration-line: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z" fill-rule="evenodd"></path></svg></a>4) Usage</h1><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">This has been built using Python 3.6.8 version.</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Download the source code and install the following packages:</p><div class="highlight highlight-source-batchfile" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">C:\Users\thund\Source\Repos\stock-prediction-deep-neural-learning<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">></span>pip list
Package Version
-------------------- ---------
absl-py 0.8.0
astor 0.8.0
astroid 2.3.3
backcall 0.1.0
certifi 2020.6.20
chardet 3.0.4
colorama 0.4.1
cycler 0.10.0
decorator 4.4.0
Django 2.2.6
gast 0.2.2
google-pasta 0.1.7
graphviz 0.13.2
grpcio 1.23.0
h5py 2.10.0
idna 2.10
image 1.5.27
imageio 2.6.1
imbalanced-learn 0.5.0
imblearn 0.0
ipython 7.8.0
ipython-genutils 0.2.0
isort 4.3.21
jedi 0.15.1
joblib 0.14.0
Keras 2.3.1
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.0
kiwisolver 1.1.0
lazy-object-proxy 1.4.3
lxml 4.5.1
Markdown 3.1.1
matplotlib 3.1.1
mccabe 0.6.1
multitasking 0.0.9
networkx 2.4
numpy 1.17.2
opencv-python 4.1.1.26
opt-einsum 3.1.0
pandas 0.24.0
pandas-datareader 0.5.0
parso 0.5.1
pickleshare 0.7.5
Pillow 6.2.0
pip 20.1.1
prompt-toolkit 2.0.10
protobuf 3.9.2
pydot 1.4.1
Pygments 2.4.2
pylint 2.4.4
pyparsing 2.4.2
python-dateutil 2.8.0
pytz 2019.2
PyWavelets 1.1.1
PyYAML 5.1.2
requests 2.24.0
requests-file 1.5.1
requests-ftp 0.3.1
scikit-image 0.16.2
scikit-learn 0.21.3
scipy 1.3.1
seaborn 0.9.0
setuptools 41.2.0
six 1.12.0
sqlparse 0.3.0
tensorboard 2.0.0
tensorflow 2.0.0
tensorflow-estimator 2.0.1
tensorflow-gpu 2.0.0
termcolor 1.1.0
traitlets 4.3.3
typed-ast 1.4.0
urllib3 1.25.9
wcwidth 0.1.7
Werkzeug 0.16.0
wheel 0.33.6
wrapt 1.11.2
xlrd 1.2.0
yfinance 0.1.54</pre></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">Then edit the file "stock_prediction_deep_learning.py" to include the Stock you want to use and the relevant dates and execute:</p><div class="highlight highlight-source-batchfile" style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px;"><pre style="background-color: #f6f8fa; border-radius: 6px; box-sizing: border-box; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;">python stock_prediction_deep_learning.py</pre></div></div><div><br /></div><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">All source code can be found here:</p><p style="background-color: white; box-sizing: border-box; color: #24292e; font-size: 16px; margin-bottom: 16px; margin-top: 0px;">- <a href="https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning" style="background-color: transparent;">https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning</a></p>Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-78006325285752049372020-05-11T18:56:00.002+02:002020-07-19T16:51:35.162+02:00Deploy a Containerized Asp.net Core 3.1 Web Api to Kubernetes<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeqRjm56Cj_Eu5-1nBZ1qtjFT19avSaqe2NCsL20LBRgbJwXYtQb0iAAxoVFW5n1JHz4Ue1DOwVxMBOWmRvKO8xQnEzdN3JHzfnnczJ93zkHtnCLYPW6iLyCoUBE-7TApGdkrwG9gEV4k/s1600/docker.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="131" data-original-width="307" height="85" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeqRjm56Cj_Eu5-1nBZ1qtjFT19avSaqe2NCsL20LBRgbJwXYtQb0iAAxoVFW5n1JHz4Ue1DOwVxMBOWmRvKO8xQnEzdN3JHzfnnczJ93zkHtnCLYPW6iLyCoUBE-7TApGdkrwG9gEV4k/s200/docker.png" width="200" /></a></div>
It's been a while since I last wrote and it's been a very interesting journey so
far. Busy as usual, it's been hard to find the time to write something
meaningful here. This time I'm bringing you something I've been playing with for
a while and that I hope it can bring your development to another level. In this
article, I will show you how easy it is to containerize your .net core
applications into a Docker container and then deploy it to a Kubernetes cluster.
I'm also preparing my Rpi cluster to become my main Kubernetes cluster and I
will write about it once everything is up and running.
<br />
<div>
<br /></div>
<div>
<span style="font-size: x-large;"><b>Requirements</b></span>
</div>
<div>
First, you'll need the following tools and requirements (Note that I'm using
Windows 10 as my main OS, so everything will be configured around it):
</div>
<div>
<br /></div>
<div>
<b>1) Docker Desktop for Windows:</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrpFvZA8rg2MtZCS5gOdkHJGnZAZ6VHFYSH9EGYISYiHnAkbSLbftx9vP5h0RTKVul3FaSKg6xf3fe1BPf7XuLu0k46oxhI35iEZI_rAyPuEoUEsS0ouBTgfLAGbKDUSD07T3YF5OuJz0/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="530" data-original-width="760" height="446" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrpFvZA8rg2MtZCS5gOdkHJGnZAZ6VHFYSH9EGYISYiHnAkbSLbftx9vP5h0RTKVul3FaSKg6xf3fe1BPf7XuLu0k46oxhI35iEZI_rAyPuEoUEsS0ouBTgfLAGbKDUSD07T3YF5OuJz0/w640-h446/dockerdesktop.png" width="640" /></a>
</div>
<div>
At the time of this article, I'm using <b>Docker 2.2.0.5</b> which should be the latest one. You can get the latest from here: (<a href="https://docs.docker.com/docker-for-windows/install/">https://docs.docker.com/docker-for-windows/install/</a>) and follow the steps from the website.
</div>
<div>
<br /></div>
<div>
<b>2) Enable Kubernetes</b></div>
<div>
Once everything is installed, open Docker preferences and select Kubernetes.
Enable all the options there and restart Docker. That should bring back Docker
and Kubernetes and you should see them online as in the picture below:
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0atewDquSoZAdrBs-CMR5xgLjGs3A-pSDvpoppKNqBzBka0LJt5Q2sY-I0rZrDGFg1-iyLrUJOzyzAW8HDnKkXUVe_MyrksbfhdawQBmDCvWxl6xZidZrPF3tUMV_Y22L_8CYWs4E_DE/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="612" data-original-width="1040" height="376" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0atewDquSoZAdrBs-CMR5xgLjGs3A-pSDvpoppKNqBzBka0LJt5Q2sY-I0rZrDGFg1-iyLrUJOzyzAW8HDnKkXUVe_MyrksbfhdawQBmDCvWxl6xZidZrPF3tUMV_Y22L_8CYWs4E_DE/w640-h376/enabledocker.png" width="640" /></a>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>3) Enable File Sharing</b></div>
<div>
You need to make sure you enable File Sharing so the container can see your local drive.
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqouLRpIhEU4156cWqy6MRxRMZp6PiDtNCUR6YlzWf73BBYvYJbNoloxRs-oFNvdtHgWb5mCbYJTSinmVmA-FYXlcu1dZKyLDQ99xEz2cxcZF3u8hbVepuPEel9OLDEPt-Cvr0CWwjsAM/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="612" data-original-width="1040" height="376" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqouLRpIhEU4156cWqy6MRxRMZp6PiDtNCUR6YlzWf73BBYvYJbNoloxRs-oFNvdtHgWb5mCbYJTSinmVmA-FYXlcu1dZKyLDQ99xEz2cxcZF3u8hbVepuPEel9OLDEPt-Cvr0CWwjsAM/w640-h376/filesharing.png" width="640" /></a>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>4) Get VS Code and Install Docker plugin (v1.1.0)</b></div>
<div>
I won't go into much detail about .net core installation as I'm guessing you
should all be able to build and run asp.net core applications locally (<a href="https://code.visualstudio.com/Download">https://code.visualstudio.com/Download</a>). In this step, you'll need to download VS Code if you haven't already and the Docker plugin which will allow us to create docker files easily as they already have a pre-built template.
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEify8UdWseZ_b-aoPX32fCCMDRyotXW_W6K-hquQJdSRLBnO41H1-T8aJNQp2zr5cLa5TVJOeTCayMlMh_duwrKNVB4bhas1rrHnSw0_n2a_w25F57zeT9LMNcEsprNaGYHIadHf-WaxAI/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="724" data-original-width="1525" height="304" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEify8UdWseZ_b-aoPX32fCCMDRyotXW_W6K-hquQJdSRLBnO41H1-T8aJNQp2zr5cLa5TVJOeTCayMlMh_duwrKNVB4bhas1rrHnSw0_n2a_w25F57zeT9LMNcEsprNaGYHIadHf-WaxAI/w640-h304/dockervs.png" width="640" /></a>
</div>
<div>
<br /></div>
<div>
<b>5) Get VS Code Kubernetes plugin (v1.2.0)</b></div>
<div>
Install the plugin to generate the YAML files that will be needed to configure the cluster. This requires the YAML component and at the time of this article, I was using version 0.8.0.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl2mr0JJjRgRi-woJES6SIfbf_arlCUNNNa5IAJvwUedDzLr1xyqEqm2PO2i5_571ke8zckuPYnZb5_2ykn6XSrQoIK07rhYVa0Jvc6QfxmOmVZZLA5XebGCXrNmleK68_kUlqyruA6oQ/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="652" data-original-width="1479" height="282" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl2mr0JJjRgRi-woJES6SIfbf_arlCUNNNa5IAJvwUedDzLr1xyqEqm2PO2i5_571ke8zckuPYnZb5_2ykn6XSrQoIK07rhYVa0Jvc6QfxmOmVZZLA5XebGCXrNmleK68_kUlqyruA6oQ/w640-h282/kubernetesextension.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>6) API Testing</b></div>
<div>
In order to test the API, I recommend using <b>Swagger </b>(<a href="https://swagger.io/tools/open-source/getting-started/">https://swagger.io/tools/open-source/getting-started/</a>) as it will easily expose the API functions and arguments that you can test.
Alternatively, you can use Postman (<a href="https://www.postman.com/downloads/">https://www.postman.com/downloads/</a>) which is also a great tool for this.
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><span style="font-size: x-large;">Dockerizing your Web API</span></b>
</div>
<div>
Now we are ready to go to our Web API project and create a docker file. Note that I'm using a complex project for this exercise and not just a simple hello world. This API has additional complex requests and it also requires a
Database. It also runs a few services and exposes most of the functionality via Swagger UI.
</div>
<div>
<br /></div>
<div>
Go to the folder of your project and type "<b>code .</b>" to launch vscode on the project. Then go to <b>view -> command palette</b> and type "<b>docker add</b>" as in the picture below and select "<b>Docker Add Files to Workspace</b>":
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeC7YVKjgwQhqiuQHzxaxDUoNY1N-dekQ5s1DSJoYMsydAKDNpBjEMKwrd74V4SNG3IAAs0dutbJgektLWZ2q8PzuhnnXpvNygnVRh4vqNtJia3wCX6n61Q_Kddxgys1FfrDkYfY473zs/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="493" data-original-width="1109" height="284" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeC7YVKjgwQhqiuQHzxaxDUoNY1N-dekQ5s1DSJoYMsydAKDNpBjEMKwrd74V4SNG3IAAs0dutbJgektLWZ2q8PzuhnnXpvNygnVRh4vqNtJia3wCX6n61Q_Kddxgys1FfrDkYfY473zs/w640-h284/dockeradd.png" width="640" /></a>
</div>
<div>
<br /></div>
<div>
Once you add the template. it will ask you about the following details:
</div>
<div>
- <b>Application platform:</b> ASP.NET Core.</div>
<div>
- <b>Operating System (container)</b>: Linux</div>
<div>
- <b>Ports to open</b>: 80 and 443</div>
<div>
<br /></div>
<div>
The example used is a bit more complex and it includes several dependencies that need to be copied across during the creation of the container. Below you can find a dependency diagram which shows the project structure:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVvB4VJvg46psiwwpURUhBw0HxuJ-f6WwJkhgEWcFpCtXlTF1KuvK0PkQBpP8IsAyEVbMck37ZXHqk9yzXI45qa4g1y8XKidRBlqQ2iKuMGh7JdLCjbayxaguOADLbeUfcB_0-9UGfcBs/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="250" data-original-width="897" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVvB4VJvg46psiwwpURUhBw0HxuJ-f6WwJkhgEWcFpCtXlTF1KuvK0PkQBpP8IsAyEVbMck37ZXHqk9yzXI45qa4g1y8XKidRBlqQ2iKuMGh7JdLCjbayxaguOADLbeUfcB_0-9UGfcBs/w640-h178/dependencies.png" width="640" /></a></div>
<div>
The application consists of a WebAPI that allows you to submit trading orders and it includes a service layer that performs the triggers internally. It also connects to a SQL Server DB which sits outside the container on my local machine. So, the connection string in the project needs to point to my local machine (<b>192.168.1.129</b>) plus the SQL Server port (<b>1433</b>). Sample connection string: (<b>"DefaultConnection": "user id=user; password=password;Initial Catalog=TradingPlatform;Data Source=192.168.1.129,1433;"</b>).</div>
<div>
<br /></div>
<div>
The final docker file should look as follows:</div>
<div>
<script src="https://gist.github.com/JordiCorbilla/7cf3c8d1ae153aebe178516a95ae33ce.js"></script></div>
<div>
<br /></div>
<div>
The file mentions that it will create an asp.net core 3.1 base layer and that it will switch to the working directory app exposing ports 80 and 443. The second image includes the SDK and it will copy all our source code there. And it will run the different commands to restore the dependencies and finally to run the dotnet build command on our solution.</div>
<div>
<br /></div>
<div>
Now that we have our docker file, we can run it using the following commands (open a new terminal via VSCode and type "<b>docker build -t trading-platform:v1 .</b>". The output should look like:</div>
<div>
<script src="https://gist.github.com/JordiCorbilla/65a41074af9696d3911ea22c3f59eed3.js"></script>
</div>
<div>
<br /></div>
<div>
If everything works as expected, the API should be in the docker container and we should be able to see that everything is running as expected (using command <b>docker images</b>):</div>
<div>
<script src="https://gist.github.com/JordiCorbilla/7739d86b21d4508a971ec38159e014af.js"></script></div>
<div>
<br /></div>
<div>
Now we just need to run our container (<b>docker run -it --rm -p 8080:80 trading-platform:v1</b>) and test that everything is working correctly. Note that the exposed port is 8080.</div>
<div>
<script src="https://gist.github.com/JordiCorbilla/1d23a485230e999215881d98a1baae28.js"></script></div>
<div>
<br /></div>
<div>
As you can see in the image below, the API is up and running and I can explore it via Swagger UI on <b>localhost:8080/swagger/api/index,html </b>which is the port we have exposed through our container:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlcrv3WkXShNnVpsNBV4cCQCxRZyaNztIi9sNAuvXlRhZpa3ZbvY3yDSFOJ31FzZsvU9R3Jbv9765cyLT6rTUFFafRDV_fbWCSBa0REFUEMn3DQCJIBrrQNGxaCCJpwHwIyXSSz0rYb0k/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="593" data-original-width="1574" height="242" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlcrv3WkXShNnVpsNBV4cCQCxRZyaNztIi9sNAuvXlRhZpa3ZbvY3yDSFOJ31FzZsvU9R3Jbv9765cyLT6rTUFFafRDV_fbWCSBa0REFUEMn3DQCJIBrrQNGxaCCJpwHwIyXSSz0rYb0k/w640-h242/webapisample.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<span style="font-size: x-large;"><b>Deploying it to Kubernetes</b></span></div>
<div>
<br /></div>
<div>
Now that we have our docker container with an ASP.NET Core Web API that talks to a SQL Server DB and that will now be deployed to a Kubernetes cluster. Your Kubernetes cluster should already be up and running if you had followed the steps above during the Docker installation.</div>
<div>
<br /></div>
<div>
Check that your Kubernetes context is switched to docker-desktop:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg71ohiN_26_muz8B3B6cDuHOIdCtAebWMrp74hrnnpxXg33EVSpRUw9csSGza5PqaUcJblJYpTmuQjZcmczpjMFU_DiNDP-HDHIj95rZQ1P9jvMxRHzrpJk8KK3LFHWSFSU4G9Yy4W5AQ/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="153" data-original-width="422" height="145" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg71ohiN_26_muz8B3B6cDuHOIdCtAebWMrp74hrnnpxXg33EVSpRUw9csSGza5PqaUcJblJYpTmuQjZcmczpjMFU_DiNDP-HDHIj95rZQ1P9jvMxRHzrpJk8KK3LFHWSFSU4G9Yy4W5AQ/w400-h145/dockercontext.png" width="400" /></a></div>
<div>
You can check that the configuration is correct by using "<b>kubectl config get-contexts</b>":</div>
<div>
<script src="https://gist.github.com/JordiCorbilla/8e5c5aa0259b8a7f3e7306b5108d67b6.js"></script></div>
<div>
<br /></div>
<div>
This will allow us to select the group we want to work with. Note that I have additional clusters created on my local.</div>
<div>
<br /></div>
<div>
Now we need to create the deployment file (<b>deployment.yml</b>). Generate a new deployment.yml file in your folder and then via VS Code, type "<b>deployment</b>" and that will bring up the inline annotation from the Kubernetes plugin. Then fill in the gaps with the information you need to set up the cluster and pods as shown in the information below:</div>
<div>
<script src="https://gist.github.com/JordiCorbilla/a548b21aaf946ec3a64c25817e27baf1.js"></script></div>
<div>
<br /></div>
<div>
We will provide a deployment name "trading-platform-deployment", the name of the pod "trading-platform-pod" and the name of the docker container to use, which in our case is called "trading-platform:v1". We will then specify port 80 as the port of the container.</div>
<div>
If everything goes well, you should see your deployment in Kubernetes ready and also the pods. We can also see that the app is running by inspecting the logs:</div>
<div>
<script src="https://gist.github.com/JordiCorbilla/82567fbb1dff77dc35aeabf907d65e8e.js"></script></div>
<div>
<br /></div>
<div>
In order to make it publicly available, we need to provide a service layer that will give us the IP to reach the container. Now we need to generate a <b>service.yml</b> file (press cntrl+space to trigger the Kubernetes plugin and select the deployment service option to scaffold the service template):</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/JordiCorbilla/50a30d7cfab5afae54221a203eb2743f.js"></script></div>
<div>
<br /></div>
<div>
The service is linked to the Pod and we specify the port we want to expose (8080) and the port in the container (80) and also the type of service which is LoadBalancer in our case. We can now see that everything is running correctly using the following commands:</div>
<div>
<script src="https://gist.github.com/JordiCorbilla/c37188322aeb4820fe2a26f83bb3de56.js"></script></div>
<div>
<br /></div>
<div>
And presto! Now we have our API running on a Docker Container and deployed to a Kubernetes cluster and to make it more real, the project is a complex project with services and additional libraries and also with a connection to a SQL Server DB. </div>
<div>
<br /></div>
<div>
If we browse to (<a href="http://localhost:8080/swagger/api/index.html">http://localhost:8080/swagger/api/index.html</a>) we will be able to reach the API.</div><div><br /></div><div>Once completed, if you want to stop the service and pod, type:</div><div>- kubectl delete service trading-platform-service</div><div>- kubectl delete pod trading-platform-deployment-6bf776f966-8xs7r</div><div><br /></div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-4605148862181390402019-08-10T16:44:00.002+02:002020-07-19T16:51:39.924+02:00Date and Time in Console log in ASP.NET Core<div style="text-align: justify;">
One of the challenges that I found so far when moving towards <b>ASP.NET Core</b> is that the default <u>console logger</u> (built-in) that comes with it is not as verbose as I was expecting when displaying <b>Dates and Times</b>. To me this is the most obvious reason as I want to see <b>when</b> something happened:</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgChGBWKoe3Jzy5xElHZRu8peXuAc8bDsrMNtmyz6VA_9uZA3rJ8Z_vderV7NCHsMM76c9n847PRqt0BsQHiT-1TxqH4HfhVxrfJMb1k0MUvrawPeERs9ZhtGLuk_bW4c4q81pIV3U7JB0/s1600/loggerconsole.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="512" data-original-width="1220" height="267" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgChGBWKoe3Jzy5xElHZRu8peXuAc8bDsrMNtmyz6VA_9uZA3rJ8Z_vderV7NCHsMM76c9n847PRqt0BsQHiT-1TxqH4HfhVxrfJMb1k0MUvrawPeERs9ZhtGLuk_bW4c4q81pIV3U7JB0/s640/loggerconsole.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
As you can see in the image above, you can appreciate all the actions that are happening to the different controllers but there is no timestamp. This was raised as an issue with the <a href="https://github.com/aspnet/Logging/issues/483">aspnet team</a> but it was deferred for later as it was not a critical piece of functionality...I guess who defines if this is critical or not. In any case, I'm the bearer of good news with the solution to your problems.</div>
<h3 style="text-align: justify;">
<b>Serilog to the rescue!</b></h3>
<div style="text-align: justify;">
With Serilog you can easily create a template so we display the message in any way we see fit. To accomplish this, here are the different components that you need (I'm already using the latest .NET Core 3.0 Preview 7):</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Install the following packages:</b></div>
<div style="text-align: justify;">
</div>
<ol>
<li>- Serilog.AspNetCore (latest, at the time of this example, was <b>2.1.2-dev-00028</b>)</li>
<li>- Serilog.Sinks.Console (latest, at the time of this example, was <b>3.1.2-dev-00779</b>)</li>
</ol>
<span style="text-align: justify;">Add the following code to your </span><b style="text-align: justify;">Program.cs</b><span style="text-align: justify;"> file:</span><br />
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<script src="https://gist.github.com/JordiCorbilla/21f192c97b877f2b462c6675500484a7.js"></script></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
And now you will see <b>the log that you need</b>:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDsFyNFV5TVpvpkp2eiWshPj6P-zAFzrte_RktZKmbn1Gf5AGOh4LR9x3qwnrr8BJeOoQ34VPHSodCGgTtJnwFSoR8cUb1VI45W1loWGm_xuKQuCkOXLnJDNwGZq_EcqbcRn4PC_cMpbI/s1600/datetime.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="512" data-original-width="1220" height="268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDsFyNFV5TVpvpkp2eiWshPj6P-zAFzrte_RktZKmbn1Gf5AGOh4LR9x3qwnrr8BJeOoQ34VPHSodCGgTtJnwFSoR8cUb1VI45W1loWGm_xuKQuCkOXLnJDNwGZq_EcqbcRn4PC_cMpbI/s640/datetime.png" width="640" /></a></div>
<br /></div>
<div style="text-align: justify;">
Hope it helps!<br />
Jordi</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-4317929121819380172019-08-04T20:32:00.002+02:002019-08-08T08:38:58.903+02:00Enable .NET Core 3.0 Preview 7 on Visual Studio 2019<div style="text-align: justify;">
I've started migrating all my <i>ASP .NET Core 2.1</i> apps to the latest <i><b>ASP .NET Core 3.0</b></i> but as it's still marked as a preview as of July 2019, you will have to tell Visual Studio to allow these kinds of packages. So for now, to create ASP.NET Core 3.0 projects with VS2019 (version 16.2.1), you should do the following:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
</div>
<ul>
<li><b>Download .NET Core 3.0</b>: <a href="https://dotnet.microsoft.com/download/dotnet-core/3.0">https://dotnet.microsoft.com/download/dotnet-core/3.0</a></li>
<ul>
<li>I installed the 64 bits SDK for windows and the ASP.NET Core Installer:</li>
</ul>
</ul>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjp2eULil83GUIeE5jzJMG2oXh1WGE_2YYAe2aC45tmm1FjpAbQFD1twEX3CfZxxV0udQ3B7hfacoZ7R2QipPXTT6HeQh14_N3Wj09uvm1yEvNqKOWDrrR6CNdjgVP3fXGsrXuGpdODA5k/s1600/.netcorepreview.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="550" data-original-width="1363" height="257" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjp2eULil83GUIeE5jzJMG2oXh1WGE_2YYAe2aC45tmm1FjpAbQFD1twEX3CfZxxV0udQ3B7hfacoZ7R2QipPXTT6HeQh14_N3Wj09uvm1yEvNqKOWDrrR6CNdjgVP3fXGsrXuGpdODA5k/s640/.netcorepreview.png" width="640" /></a></div>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<ul>
<li><b>Enable preview releases of .NET Core SDK</b></li>
<ul>
<li>Click Tools | Options in the top menu</li>
<li>Expand Environment | .NET Core </li>
<li>Ensure that "<b>Use previews of the .NET Core SDK (requires restart)</b>" checkbox is checked</li>
</ul>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8023cX6D7YATdeWO2yflN6cY_iQVmqOLTcvyHqS0hAbONeOqFljBBUJO6k9y9HRiUSn76BHWUIW_BDlJFYoc8YdwzqrJB7ejfZbUUl_0xhN5WT7T1cqDudn-nkVF1NxMCttDbOKocndk/s1600/preview.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="433" data-original-width="744" height="372" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8023cX6D7YATdeWO2yflN6cY_iQVmqOLTcvyHqS0hAbONeOqFljBBUJO6k9y9HRiUSn76BHWUIW_BDlJFYoc8YdwzqrJB7ejfZbUUl_0xhN5WT7T1cqDudn-nkVF1NxMCttDbOKocndk/s640/preview.png" width="640" /></a></div>
<div>
<br /></div>
<div>
Once you restart Visual Studio and install all the necessary components mentioned above, you should see the following option under <b>Target framework</b> in your project:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilOuWbfh8l78fz-Qz9NRxM-izpXcIoaBm10bqLmzSDVrlPgmQpRmcAWywjDlkUZkK-81IrX1JvgrAZgU-YD_8kllpjlA4CLwDUgMaNYN5_9ShF5UrnoLCm3Zgrp2gJ3ZT63TyrYP_V8cE/s1600/projectinfo.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="358" data-original-width="811" height="281" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilOuWbfh8l78fz-Qz9NRxM-izpXcIoaBm10bqLmzSDVrlPgmQpRmcAWywjDlkUZkK-81IrX1JvgrAZgU-YD_8kllpjlA4CLwDUgMaNYN5_9ShF5UrnoLCm3Zgrp2gJ3ZT63TyrYP_V8cE/s640/projectinfo.png" width="640" /></a></div>
<div>
<br /></div>
<div>
Now, to build your project and make it work is another thing, but at least the framework is there your you to play with!</div>
<div>
<br /></div>
<div>
Happy Coding!</div>
<div>
Jordi</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-53631441724899100532018-09-16T15:50:00.001+02:002018-09-16T15:50:14.948+02:00Creating a File Server using ASP.NET Core 2.1 Static Files Middleware<div style="text-align: justify;">
One of the coolest features of <b>ASP.NET Core</b> is the ability to serve static files on HTTP requests without any<b> server-side</b> processing. This means that you can leverage this technology to build your own file server to serve static files over the wire. I managed to code a quick solution over the weekend to be able to upload/download files to one of my servers easily with this technology.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
It all started with <a href="https://twitter.com/thunderjordi/status/1039076946712453120">this</a> Tweet:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZV5-8jYJa-m1OfKrW555IEDFfVU1ZsaHzIRmu2B31MJfY9Yt9xJDXWM7Np0PS0AgdU7w_KmXeiHtEst9IwjL3riLwYeVL0OtQvYHSUrcwflp9PzmkdkUhzmKCmEpKcXCUQZ4k4BcwrvI/s1600/tweet.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="718" data-original-width="1260" height="364" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZV5-8jYJa-m1OfKrW555IEDFfVU1ZsaHzIRmu2B31MJfY9Yt9xJDXWM7Np0PS0AgdU7w_KmXeiHtEst9IwjL3riLwYeVL0OtQvYHSUrcwflp9PzmkdkUhzmKCmEpKcXCUQZ4k4BcwrvI/s640/tweet.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The issue here is that both of us spend loads of time taking pictures and videos of our daughter and end up sharing all this data via Whatsapp. By the end of the day we both have the same information but in different resolutions and qualities which makes our life quite difficult when trying to guess which picture has the highest quality for printing (we tend to print most of the pictures). So we needed something simpler and quicker where to store all this pictures and videos (keeping the highest possible quality) and that it would be easier for both of us to share.</div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: justify;">
<b>.NET Core and Ngrok to the rescue. </b></h3>
<div style="text-align: justify;">
With <a href="https://ngrok.com/">Ngrok</a>, I can easily expose one of my websites to the world via one of the ngrok tunnels. I do own a professional account with them and it really made my life much easier as I can expose whatever I need to the world without having to tinker with my router. This helps me to expose services from my Raspberry Pi's and from my Servers.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Using .NET Core Static files middleware, I was able to build a quick solution (accessible via any browser and mobile responsive) with just 300 lines of code.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The main <b>features</b> of the application are:</div>
<div style="text-align: justify;">
</div>
<ul>
<li>Multiple file uploader (up to 300 Mb).</li>
<li>Multiple file downloader.</li>
<li>Daily browsable functionality (it allows you to navigate on each day to see the list of files uploaded).</li>
<li>Thumbnail automatic generation using <a href="https://github.com/saucecontrol/PhotoSauce">MagicScaler</a>.</li>
<li>Automatic movie conversion via <a href="https://cloudconvert.com/api">CloudConvert</a> (this allows me to share videos between mobile devices as iPhones generate .mov files which cannot be played on Android devices).</li>
<li>Keep the existing quality of the file (full size). This means uploading <u>huge files</u> into the File server.</li>
<li>Cookie authentication with Policies.</li>
</ul>
<br />
<div style="text-align: justify;">
You can see the flow in the image below:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjadyGz1dyadOhhzfM5_fA_IJRLRiXJ9pzO-irpdoEJV3gfkeb01Dn1Nn0SXNKJFzjl645oL-H3Zu_zSHqa8620LYeP6x4VhVnBvczgPqRh_WW9XPMdtzxLBXuQjhSMtyvLReMuaQuL3Fo/s1600/Animation.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="893" data-original-width="984" height="580" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjadyGz1dyadOhhzfM5_fA_IJRLRiXJ9pzO-irpdoEJV3gfkeb01Dn1Nn0SXNKJFzjl645oL-H3Zu_zSHqa8620LYeP6x4VhVnBvczgPqRh_WW9XPMdtzxLBXuQjhSMtyvLReMuaQuL3Fo/s640/Animation.gif" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Sample code for the file uploader can be found below:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<script src="https://gist.github.com/JordiCorbilla/f017c2af4cb5a0a56523a80b9a27b9c1.js"></script></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
To list the files, I use the <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/file-providers?view=aspnetcore-2.1">File provider in ASP.NET Core</a> which uses the <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/static-files?view=aspnetcore-2.1&tabs=aspnetcore2x">static file middleware</a> to locate static files, in my case pictures and videos.<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/ad13dbb53429e96c055deacf8ee485f0.js"></script><br />
There are quite a lot of things to do to improve the application but now anyone who uses the application can easily upload/download pictures from one of my servers which is monitored and constantly backed up so no picture gets lost. Also segregating pictures by dates helps a lot to find the one you were looking for.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I'm really impressed with the latest release of .NET Core (note that this has been built with .NET Core 2.1 and there is already a 2.2 preview version available) as the request are really fast and you can't even notice any lag even browsing with your phone on 3G which gives a nice user experience.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
This is how it looks when browsing from my phone:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Z_G6loAD6VNoYM79FPsx20D42PC8FrfxZ2_cknZqT5_TH7mk6eaEqPQsGxntMCotsjrb0GJS5K-6g2ICLT0P2HHhGQFieFBCVzOgU3p5N6PwTSzMBX7ZFzoocvSVY1XcR0zmRQQdf3c/s1600/Screenshot_20180916-143223.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="1600" data-original-width="900" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5Z_G6loAD6VNoYM79FPsx20D42PC8FrfxZ2_cknZqT5_TH7mk6eaEqPQsGxntMCotsjrb0GJS5K-6g2ICLT0P2HHhGQFieFBCVzOgU3p5N6PwTSzMBX7ZFzoocvSVY1XcR0zmRQQdf3c/s400/Screenshot_20180916-143223.png" width="225" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPO_uXNGSRIX3GwFRGw3JWs1SrM40GJshxUH4Jc80zXzcPAZDTIaQPGWyuQxcWX9-7RWWaAB1q39tKEwzYE-icKc1WSwSItMeyR4sKbg9did3OBQFQbjKjXAlqfwpQfdghlYA9SKIPiTw/s1600/Screenshot_20180916-143244.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="900" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPO_uXNGSRIX3GwFRGw3JWs1SrM40GJshxUH4Jc80zXzcPAZDTIaQPGWyuQxcWX9-7RWWaAB1q39tKEwzYE-icKc1WSwSItMeyR4sKbg9did3OBQFQbjKjXAlqfwpQfdghlYA9SKIPiTw/s400/Screenshot_20180916-143244.png" width="225" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgk0a9zyBBB9QeHdW50nMkKcG7TIZ9W3mWtaViUNeHWKsz5IHTnM0j_JTAQiD7ZLhsQuECXlLAx_6C6CnkAVqzvZ6AM1-C95bt3BXEH3tDsPAg_AMPnAGIwBF6LJqY3bFtG7i0EGM1k7Hg/s1600/Screenshot_20180916-143153.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="900" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgk0a9zyBBB9QeHdW50nMkKcG7TIZ9W3mWtaViUNeHWKsz5IHTnM0j_JTAQiD7ZLhsQuECXlLAx_6C6CnkAVqzvZ6AM1-C95bt3BXEH3tDsPAg_AMPnAGIwBF6LJqY3bFtG7i0EGM1k7Hg/s400/Screenshot_20180916-143153.png" width="225" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Source code will be available soon on my <a href="https://github.com/JordiCorbilla">Github</a> page as I'm trying to figure out one of the issues with https redirection which still does not work correctly and without it, it doesn't make sense.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<div style="text-align: right;">
Jordi Corbilla</div>
</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-49522610217253186962018-07-01T19:50:00.000+02:002018-07-01T19:50:01.040+02:00Configure TeamCity to access private GitHub Repositories<div style="text-align: justify;">
One of the challenges I have been facing lately after moving to <b>private repositories</b> on GitHub is the ability to access them via <b>TeamCity</b>. The issue is that now the repository is <b>not</b> accessible via https and you have to find an alternative to retrieve the source code of your repository securely.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
For this task, I will show you how to use <b>GitHub Deploy keys</b> and how to configure <b>TeamCity</b> to interact with your private repository.</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQr33pSdDCFJJ8m3gesCGsghbeGshKzJwP1h9tYhBVqaFVhw0amtQtVb291WrdyAKTBSOVmxUP8SPVtPeAR4hVLKC4U1F1D4wnvdGbuVNa1pRhWseYJtINSpPcOApFUMRNbCaN6-nG4GE/s1600/tcdiagramjordicorbilla.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="356" data-original-width="781" height="290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQr33pSdDCFJJ8m3gesCGsghbeGshKzJwP1h9tYhBVqaFVhw0amtQtVb291WrdyAKTBSOVmxUP8SPVtPeAR4hVLKC4U1F1D4wnvdGbuVNa1pRhWseYJtINSpPcOApFUMRNbCaN6-nG4GE/s640/tcdiagramjordicorbilla.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The overall idea can be seen in the figure above. First, we will have to create the keys so we can place them in the required section. To generate the keys, you can just use <a href="https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/">Git Bash</a> and using the command below:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<script src="https://gist.github.com/JordiCorbilla/93d5ae67879c34fd47bca6db13962c26.js"></script></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Once finished, you will have two keys, the private and the public one.<br />
<br />
<h2>
<b><span style="font-size: large;">Installing the public key in GitHub:</span></b></h2>
The operation above should've produced 2 keys (files):</div>
<div style="text-align: justify;">
<br />
<ul>
<li>id_rsa (private key)</li>
<li>id_rsa.pub (public key)</li>
</ul>
<br />
Open the file <b>id_rsa.pub</b> or run the following command on your git bash console to copy the content of the file into the clipboard: <b>clip < ~/.ssh/id_rsa.pub</b><br />
<br />
Now, go to your private repository on GitHub and select "Settings" and then "Deploy Keys". Once there, click on "Add deploy key" and paste the content of the file you've opened before / copied into the clipboard.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhljHdxTLw-vyOqk_Qt1cwBl-2sNQ9wBMRGIUqAiOO6ws8aUE6ruqAqfCJYWIZ73Ds-B9XUaylmpUATy5zGqwKL7I5rh4lOzBbjHXoJXz4LhkpnpkmJsdPu3bFdRcxUfv-VYAGmoEWfhtg/s1600/deploy-key.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="595" data-original-width="1000" height="380" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhljHdxTLw-vyOqk_Qt1cwBl-2sNQ9wBMRGIUqAiOO6ws8aUE6ruqAqfCJYWIZ73Ds-B9XUaylmpUATy5zGqwKL7I5rh4lOzBbjHXoJXz4LhkpnpkmJsdPu3bFdRcxUfv-VYAGmoEWfhtg/s640/deploy-key.png" width="640" /></a></div>
<br />
Once completed, you should see something like the image below (note that the image below shows that the key has been already used):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9mO-LInSRv1gmqgWEyzVF44VIH53w5s-4bUUs00FrmxqITdemnRXOYveFzVV7w6geYV0agh7GQxUnsFgyWhkYr2GCh5gEFXNMxCr4yXRaWxlNQqzdYA8NrMhwn7BVLwo2gBd_ztWCd7o/s1600/deploy-keys2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="356" data-original-width="996" height="228" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9mO-LInSRv1gmqgWEyzVF44VIH53w5s-4bUUs00FrmxqITdemnRXOYveFzVV7w6geYV0agh7GQxUnsFgyWhkYr2GCh5gEFXNMxCr4yXRaWxlNQqzdYA8NrMhwn7BVLwo2gBd_ztWCd7o/s640/deploy-keys2.png" width="640" /></a></div>
<br />
<h2>
<b><span style="font-size: large;">Installing the private key in TeamCity:</span></b></h2>
The following operations have been done to the latest version of TeamCity at the time of the publication of this article (2018.1 build 58245). I tried version 2017 initially and the configuration didn't work (just so you know if you are still on any version prior to 2018.1):<br />
<br />
Click on your project overview and click on "Edit project Settings". Select "SSH Keys" and click "Upload SSH Key" button to upload your <b>id_rsa</b> file:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWOzl6BtnAPk1Ahw9DYMwx6g6kXQWtfTERLdL5mSdlAR1h-gpqlwPj_K-UZwK8DcX15xQelMvUOVmXUvBJsTk7KeOtOm22pXbGRs_vSSU4Rl6YzmFqGESyZZg3gKBFqY6_WeRCtE4egIs/s1600/tcprivate.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="564" data-original-width="1552" height="232" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWOzl6BtnAPk1Ahw9DYMwx6g6kXQWtfTERLdL5mSdlAR1h-gpqlwPj_K-UZwK8DcX15xQelMvUOVmXUvBJsTk7KeOtOm22pXbGRs_vSSU4Rl6YzmFqGESyZZg3gKBFqY6_WeRCtE4egIs/s640/tcprivate.png" width="640" /></a></div>
<br />
Now the SSH key will be available in your VCS Root. Now go to your build step and add a Git VCS Root that will pull the source code from the repository. The parameters that you have to configure are as follow:<br />
<br />
<ul>
<li><b>VCS Root Name</b>: Name of your VCS.</li>
<li><b>Fetch URL</b>: URL of your repository in format git (not in https format as it will not be available because the repository is private). In this case you will have to change the https URL by this other git one as shown in the sample below:</li>
<ul>
<li><a href="https://github.com/JordiCorbilla/easyTimeSheets">https://github.com/<span style="color: red;">JordiCorbilla</span>/<span style="color: magenta;">easyTimeSheets</span></a> with:</li>
<li>git@github.com:<span style="color: red;">jordicorbilla</span>/<span style="color: magenta;">easytimesheets</span>.git</li>
<li>where user name and repository need to be part of the URL</li>
</ul>
<li><b>Default branch</b>: refs/heads/master</li>
<li><b>Authentication method</b>: Uploaded Key</li>
<li><b>Username</b>: empty (don't type anything here)</li>
<li><b>Uploaded Key</b>: id_rsa (is the one that I've just uploaded)</li>
<li><b>Password</b>: type the secret word you have configured in your private key if any.</li>
</ul>
<br />
If you now test the connection, it should be successful:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiU5K_JGWk1VzKA0c6JTt7-WMp0aX1VvA8d9PMD50UJab5qLHZI32gGEdVga8ItgJ8DKf7V2HLL7FlIwHkn78qoAjdhBab0QuxgiixVuKHbp6fRgvbjQmz3q3nPtdCeZVHN72QfMzd2r24/s1600/testconnection.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="762" data-original-width="1436" height="338" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiU5K_JGWk1VzKA0c6JTt7-WMp0aX1VvA8d9PMD50UJab5qLHZI32gGEdVga8ItgJ8DKf7V2HLL7FlIwHkn78qoAjdhBab0QuxgiixVuKHbp6fRgvbjQmz3q3nPtdCeZVHN72QfMzd2r24/s640/testconnection.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
If you have a look at your project, you will see that the project is successfully connecting to your repository and pulling out the changes that are pending to be implemented in your pipeline:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPtFhB8WyU1tT02hYBHS6eZuo_0o0ZvXJl-dfvqZp3p9YL3ozkPGvwDSImt_U2t2A0emyoIvGEcwjLdBs_on85BEAVUUXAFvahRUdtdMHYt1ZHpNOfV4fITC54IzTYzmpQHQJmoad3GvI/s1600/deploy.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="356" data-original-width="1550" height="146" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPtFhB8WyU1tT02hYBHS6eZuo_0o0ZvXJl-dfvqZp3p9YL3ozkPGvwDSImt_U2t2A0emyoIvGEcwjLdBs_on85BEAVUUXAFvahRUdtdMHYt1ZHpNOfV4fITC54IzTYzmpQHQJmoad3GvI/s640/deploy.png" width="640" /></a></div>
I hope you find it useful as I have spent quite a lot of time just trying to find the right approach.<br />
<br />
<div style="text-align: right;">
Jordi</div>
</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-73762400883350051182018-04-15T19:18:00.000+02:002018-04-15T19:18:10.108+02:00Detecting Ajax Requests In ASP.NET MVC 5<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLAQcE5tu0ZicPMvgDwnPTt3ZbqGPsvfQXZCUmSq5y9ygY4iaZOlMRgO_mcyHXmEuVM45fWz0uxui0fec00Pq35RDxeJKGEx-IblxhgMww7nshUXmIZVfG5rr7xwB4mMzbVudYziKE1vs/s1600/request.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="42" data-original-width="239" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLAQcE5tu0ZicPMvgDwnPTt3ZbqGPsvfQXZCUmSq5y9ygY4iaZOlMRgO_mcyHXmEuVM45fWz0uxui0fec00Pq35RDxeJKGEx-IblxhgMww7nshUXmIZVfG5rr7xwB4mMzbVudYziKE1vs/s1600/request.png" /></a></div>
Security is one of my major concerns nowadays and it is quite common that someone will try to browse to a particular URL given any chance. To avoid this, we can detect whether the request came from an <u>Ajax request</u> or from a <u>normal browser request</u>.<br />
<br />
Within <b>ASP.NET MVC5</b> applications is quite easy to check if a request is being made via AJAX through the extension named <a href="https://msdn.microsoft.com/en-us/library/system.web.mvc.ajaxrequestextensions.isajaxrequest%28v=vs.118%29.aspx">IsAjaxRequest</a>() method that is available on the <b>Request </b>object. The IsAjaxRequest() actually works by simply performing a check for the <b>X-Requested-With</b> header.<br />
<br />
The example below will show you how wrap this functionality inside an Action Filter so you can add this feature to any controller you want to perform this check upon.<br />
<br />
Creating the <b>Action Filter</b>:<br />
<script src="https://gist.github.com/JordiCorbilla/78fd3c474c4bf40e364fed20d3f7172a.js"></script><br />
Then just use the attribute in any of the actions you want to perform this operation:<br />
<script src="https://gist.github.com/JordiCorbilla/b976ada64db7df1f414c2b54eff2a7d2.js"></script><br />
And now, if you try to browse that action from a normal request you will get the following error:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0jAMOGtduae3Ge0hYDrirZjxJ8_6C3Jd4QRF_lu5VHAYssKJ23TzILSTVew5jA-jFQGvtrdvbuJ1-ilip2CYCnfkhX71HMal58WoJS7r-bse40fJba6agKAW9owP_aFYMmoxbazXcJgw/s1600/err.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="237" data-original-width="1220" height="123" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0jAMOGtduae3Ge0hYDrirZjxJ8_6C3Jd4QRF_lu5VHAYssKJ23TzILSTVew5jA-jFQGvtrdvbuJ1-ilip2CYCnfkhX71HMal58WoJS7r-bse40fJba6agKAW9owP_aFYMmoxbazXcJgw/s640/err.png" width="640" /></a></div>
<br />
Now you can make your site more secure without worrying about someone inadvertently clicking to the action.<br />
<br /></div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-49081505696301908062017-09-25T00:03:00.001+02:002017-09-25T21:40:23.163+02:00Make your Delphi applications pop with Font Awesome!<div style="text-align: justify;">
Yes you heard right, <b>Font Awesome</b>? You can use their icons to make your desktop applications pop. Nowadays I use it to make my websites look nicer and without having to worry about finding icons for my apps and edit them and so on. Font Awesome is one the smartest things you can use to make your applications pop.<br />
<br />
<h2>
<span style="font-size: x-large;">Use Font Awesome icons in Desktop apps</span></h2>
First <a href="http://fontawesome.io/#modal-download">download Font Awesome</a> and install it in your computer. At the time of this article I was using version 4.7.0 so I downloaded font-awesome-4.7.0.zip and installed the <b>FontAwesome.otf</b> file on my <b>Windows 10</b> machine:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFYKyC1Y125cn4YgO0U3BnjwJEnf6FqXr1agYjtAme1syYMmZ3x5drw0Sb2E5cLqR_Qe5A7for-W0OcwOtXgS-a4CsUyYY0Nn-7q_UUvmBQwqeWhhoFXrmQ2UOOdqSApw_-odei0ABZeo/s1600/install.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="629" data-original-width="1219" height="330" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFYKyC1Y125cn4YgO0U3BnjwJEnf6FqXr1agYjtAme1syYMmZ3x5drw0Sb2E5cLqR_Qe5A7for-W0OcwOtXgS-a4CsUyYY0Nn-7q_UUvmBQwqeWhhoFXrmQ2UOOdqSApw_-odei0ABZeo/s640/install.png" width="640" /></a></div>
<br />
<br />
Font Awesome provides a <a href="http://fontawesome.io/cheatsheet/">cheatsheet </a>that can be used to copy and paste the icons directly in your app so you don't have to worry about memorising any particular code to make the icon appear:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLTEaXdpv-V0m3cXK8Vb-G12cx-_wH97iuJHx2ltIouYxQ0UzAYJKeTR3JR9gWsafYGcoAiOC80u1ZzSNnF6aqwleZVnY0dRePRLnAA1rFC24X6kxQwZve__uroiXCIPRBtEAGniVHpB0/s1600/cheatsheet.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="814" data-original-width="1172" height="444" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLTEaXdpv-V0m3cXK8Vb-G12cx-_wH97iuJHx2ltIouYxQ0UzAYJKeTR3JR9gWsafYGcoAiOC80u1ZzSNnF6aqwleZVnY0dRePRLnAA1rFC24X6kxQwZve__uroiXCIPRBtEAGniVHpB0/s640/cheatsheet.png" width="640" /></a></div>
<br />
<br />
Nowadays I use the common approach where I buy certain icons or draw them myself using Photoshop (although this second option is quite time consuming and I only do it when I want to achieve the best results).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHxXjrYm-ZGtJAdXuOL4-YIjqrTD6E0FvR2x7_LYT8MKRPsfP4GMRDiI7orm7J1MC_GhGGV3lbHNnKBlgQWh5rpSEaAkZb0oDAQhFGo52MXJZgCFaydzDU9FSna_CksAubzwbxChmGk0k/s1600/icons.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="389" data-original-width="805" height="309" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHxXjrYm-ZGtJAdXuOL4-YIjqrTD6E0FvR2x7_LYT8MKRPsfP4GMRDiI7orm7J1MC_GhGGV3lbHNnKBlgQWh5rpSEaAkZb0oDAQhFGo52MXJZgCFaydzDU9FSna_CksAubzwbxChmGk0k/s640/icons.png" width="640" /></a></div>
<br />
I'm sure you are all familiar with this approach, you add your icon in bmp format into one <b>ImageList </b>component, then link the <b>ImageList </b>to the button and select the icon index so it appears in your button as displayed in the image above. The problem arises when you want to have different sizes of that button as you will have to have different icon sizes to match it and so on.<br />
<br />
So one of the things that I tend to do now in my applications and that makes it look a bit more standard (in terms of user experience as the user sees the same type of icon throughout the application) is by using Font Awesome:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6qlTZaghT_QxlaWlfvS-dYJIHlGLWnhAW9Vd0ML2MCYHtFxgErtgqBUCzkIw8bflaXy0KcFzJ_CcnmqYvTiqUtr9HsMIR5b3ZpH_uPliKmvDDvMOZJmmrd_BkVFcyswLalKTof616gbg/s1600/FontAwesome.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="780" data-original-width="1245" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6qlTZaghT_QxlaWlfvS-dYJIHlGLWnhAW9Vd0ML2MCYHtFxgErtgqBUCzkIw8bflaXy0KcFzJ_CcnmqYvTiqUtr9HsMIR5b3ZpH_uPliKmvDDvMOZJmmrd_BkVFcyswLalKTof616gbg/s640/FontAwesome.gif" width="640" /></a></div>
<br />
The animation above displays how to replace one of you icons with a Font Awesome icon easily:<br />
<br />
<ul>
<li>Locate the icon you want in the Font Awesome cheat-sheet.</li>
<li>Copy the icon image (not the Unicode code).</li>
<li>Locate the component you want to add the icon to.</li>
<li>Select Font Awesome font.</li>
<li>Paste the icon in the caption or text zone.</li>
<li>Adjust size to your needs.</li>
</ul>
<br />
In the images below you can compare the before/after and you'll see that the difference is noticeable:<br />
<br />
<b>Before (mixture of icons in bmp format plus some png images made with Photoshop):</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyspCOxIkzxzny-grQvESUcCIl8N71bHH4VhzKdGc3Nli3nHFrN-JkSGX0Wpm3Lhk9AJNCsthU6diaTx405J6blKStBAJc3_4Sb58kQsuECPS6TLPw1kl0pzLlzSSnWJKrgT46RuIFtuY/s1600/before.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="390" data-original-width="1252" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyspCOxIkzxzny-grQvESUcCIl8N71bHH4VhzKdGc3Nli3nHFrN-JkSGX0Wpm3Lhk9AJNCsthU6diaTx405J6blKStBAJc3_4Sb58kQsuECPS6TLPw1kl0pzLlzSSnWJKrgT46RuIFtuY/s640/before.png" width="640" /></a></div>
<br />
<br />
<b>After (Font Awesome fonts replacing all the icons):</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJ6KPgNgt7iS-LScz5RWru3QVBbJrJ3k1SqmFx6XPO-TovywtR50ZylHWMhdxDqjQlw0zaAGadu9ig5k4sB7e6ed8E-BRubxXlijDPMqP1mZy7lXMEAwGHWgAaEeZUZ1ux-Ut3FTsaYB0/s1600/after.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="391" data-original-width="1249" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJ6KPgNgt7iS-LScz5RWru3QVBbJrJ3k1SqmFx6XPO-TovywtR50ZylHWMhdxDqjQlw0zaAGadu9ig5k4sB7e6ed8E-BRubxXlijDPMqP1mZy7lXMEAwGHWgAaEeZUZ1ux-Ut3FTsaYB0/s640/after.png" width="640" /></a></div>
<br />
Notice that <b>now I can even include icons where there should only be text</b>! so using this way I can compose my headers in a nicer way and include a very descriptive icon.<br />
<br />
You will need Font Awesome font installed on your machine or target machine in order to take advantage of this functionality. I've used the latest Delphi 10.2 Tokyo on this one if anyone was wondering about it.<br />
<br />
The example above is for VCL only and it should also for for FMX applications.<br />
<br />
<b>Example with FMX:</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0d7SuK-VW23Nn721-GBy-_0rlZGLYc75IFnaW49dNw0RbqKgTj9MZ29GvcAi2bq3epAXkiU-bbmAa692-gXg_-8XrdND4scVaAe6jOFtR2Clx0q2lYCmkbdszSKXOCU4IF7_8fEWOxEM/s1600/firemonkey.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="746" data-original-width="944" height="504" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0d7SuK-VW23Nn721-GBy-_0rlZGLYc75IFnaW49dNw0RbqKgTj9MZ29GvcAi2bq3epAXkiU-bbmAa692-gXg_-8XrdND4scVaAe6jOFtR2Clx0q2lYCmkbdszSKXOCU4IF7_8fEWOxEM/s640/firemonkey.png" width="640" /></a></div>
<br />
<br />
<div style="text-align: right;">
<b>Jordi</b></div>
<div style="text-align: right;">
Embarcadero MVP</div>
</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com7tag:blogger.com,1999:blog-7022755517551007355.post-88038130167026035342017-07-27T22:29:00.001+02:002024-02-11T23:27:24.250+01:00JSON RTTI Mapper with Delphi<div style="text-align: justify;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIJKez1ENvRvw-bm2-Sow3v7rdh0lAr70vL_yQkX3y2y2j-5PwTs_cUgOfWXscwSTS2CF1rbycXMCdkhkBD84E1UbG5BB3iYm-oTgsgb6oP_LjNVm-ETix1h_See9gjeBg9k2MWbMtn_mKZ3Xueu13BWBIdS67RLl9q8nDm1QJtCVkljyN9wr8blxpOvA/s1024/DALL%C2%B7E%202024-02-11%2022.26.41%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_JSON%20RTTI%20Mapper%20with%20Delphi_.%20The%20image%20should%20visually%20represent%20the%20concept%20of%20mapping%20JSON%20data%20t.webp" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1024" data-original-width="1024" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIJKez1ENvRvw-bm2-Sow3v7rdh0lAr70vL_yQkX3y2y2j-5PwTs_cUgOfWXscwSTS2CF1rbycXMCdkhkBD84E1UbG5BB3iYm-oTgsgb6oP_LjNVm-ETix1h_See9gjeBg9k2MWbMtn_mKZ3Xueu13BWBIdS67RLl9q8nDm1QJtCVkljyN9wr8blxpOvA/w200-h200/DALL%C2%B7E%202024-02-11%2022.26.41%20-%20A%20simple,%20engaging%20image%20for%20a%20blog%20post%20titled%20_JSON%20RTTI%20Mapper%20with%20Delphi_.%20The%20image%20should%20visually%20represent%20the%20concept%20of%20mapping%20JSON%20data%20t.webp" width="200" /></a></div>
One of the patterns that I have observed a lot during the time I have been playing with <b>JSON </b>streams is that I create my object based on the JSON stream and then I set the properties of that particular object manually so I can work with it rather than with the JSON object itself. </div>
<div style="text-align: justify;">
<br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div>
<div style="text-align: justify;">
Let's observe the following example. Imagine that we have the following JSON stream:</div>
<div style="text-align: justify;">
<br /></div>
<script src="https://gist.github.com/JordiCorbilla/ba32f8202306203548f21fdeb7c896c2.js"></script>
<br />
<div style="text-align: justify;">
As you can see this JSON represents a list of Employees and each employee has the properties <i>Name, Surname, Age and Address</i>. So if we want to hold this in a <b>TList<T></b> then we will have to create a class <b>TEmployee </b>with those properties and then manually assign each JSON parameter to each property like the example below:</div>
<div style="text-align: justify;">
<br />
<script src="https://gist.github.com/JordiCorbilla/66dc8fe1fb3b483c0a6574cd2791a74a.js"></script>
The code is quite straight forward. We need to loop through the JSON array and populate the list.<br />
<br />
If you look closely, you will see that basically <b>we are mapping a field</b> in the JSON object that it's called "name" to an object property called "Name". So to make it simpler this would literally be something like this:<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/bcdf0d9059b41d4a1bb779df5fa2c02a.js"></script>
Any mapper out there does <b>one simple job</b> and it's the job of mapping one field from one source to another.<br />
<br />
So the question here is how to achieve this in a more clever way? Easy, let's use <a href="http://docwiki.embarcadero.com/RADStudio/Tokyo/en/RTTI_directive_(Delphi)">RTTI </a>to map those properties!<br />
<br />
Using the methods <a href="http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/delphivclwin32/TypInfo_SetStrProp@TObject@PPropInfo@string.html">TypInfo.SetStrProp</a> and <a href="http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/delphivclwin32/TypInfo_GetPropList@PTypeInfo@PPropList.html">TypInfo.GetPropList</a> you can easily explore and the list of <b>published </b>properties of your class and set the value of them. To make use of the RTTI capabilities, you will have to move those properties to the <b>published </b>section of the class so they are visible through the RTTI.<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/92ca0e8f288a7fd8179f31e05d32e574.js"></script>
Now you know how to use the RTTI to read the list of published properties and set them to a specific value. These examples have been coded with Delphi 10.2 Tokyo and you can find part of the mapper in one of the projects I'm currently working on: <a href="https://github.com/JordiCorbilla/COCAnalytics">COCAnalytics</a>.<br />
<br />
There are many libraries out there that do amazing things with JSON so it's up to you to explore them. At least now you know how to map using the RTTI.<br />
<br />
Happy coding!.<br />
<br />
<div style="text-align: right;">
Jordi</div>
<div style="text-align: right;">
Delphi MVP.</div>
</div>
<br />Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-86035603652668550862017-07-25T23:04:00.001+02:002017-07-25T23:04:53.863+02:00Writing quality code with NDepend v2017<div style="text-align: justify;">
The new version of <b>NDepend v2017</b> has totally blown my mind. I can't stop exploring the new and enhanced Dashboard with features like <b>Technical debt estimation</b>, <b>Quality Gates</b>, <b>Rules</b> and <b>Issues</b>. In this post I will try to summarise what's <a href="http://www.ndepend.com/ndepend-v2017">new</a> with <b>NDepend v2017</b> and how to use these new features to write <u>quality code</u>.<br />
<br />
I'm sure that you have experienced the feeling when you start typing code and after few weeks down the project you don't really know If what you have designed and coded is actually good or bad (by bad I mean that it's in some way <i>rigid</i>, <i>fragile </i>or <i>non-reusable</i>). I'm an advocate of Continuous Integration so I do have loads of metrics that help me identify broken windows or code smells easily in my code during the check-in stage. All these metrics encompass standards such as <i>Design, Globalisation, Interoperability, Mobility, Naming, Performance, Portability, Security, Usage</i> and so on. But none of them give a global rating that I could easily use to check if my project is actually good or bad.<br />
<br />
<h2>
<b><span style="font-size: large;">Enhanced Dashboard</span></b></h2>
This new version contains a new set of application metrics that really improved the overall quality of the product. I will start integrating this as part of my release procedure as it gives a really good grasp of the status of the project from the coding side.<br />
<br />
Here is a quick sneak peek of the new dashboard:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWf4zw8hwCHKJ7AA6EMH2U6o9hxY5WVahAtZIpMrGolHDihEvBPyt6PuUxY7Tdb34s8AT6nvvnwZZo_W8EL_waEz8M_UtAb6kCvmxQGX10fL81u-ni3A-GzTpaSfwjfE3cz4oxCwpol0I/s1600/NDependMetrics.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="588" data-original-width="956" height="393" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWf4zw8hwCHKJ7AA6EMH2U6o9hxY5WVahAtZIpMrGolHDihEvBPyt6PuUxY7Tdb34s8AT6nvvnwZZo_W8EL_waEz8M_UtAb6kCvmxQGX10fL81u-ni3A-GzTpaSfwjfE3cz4oxCwpol0I/s640/NDependMetrics.png" width="640" /></a></div>
<br />
The aspects I'm most interested in and that I will delve into detail are the following ones:<br />
<br />
<h2>
<b><span style="font-size: large;">Technical Debt Estimation</span></b></h2>
This new feature is a <b>MUST </b>for me. Just analyse your project with <b>NDepend v2017</b> and let it give you the percentage of technical debt according to the rules that you have configured in your project. After every analysis you can see the trend and act accordingly using this metric:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4ZP5BToeXad0jt8P6VpUuf1C_CfPQQWAWbxphwhl-EqrtqhBZObOzlx5MMKKeIvdzXfVN4fff9NIO7l18MidXzTHvWk7m6KP6NjrMeWNYLr3_gjmBhp8d_qhBn2YAnkMi2FG5-6Ij9JM/s1600/debt.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="226" data-original-width="261" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4ZP5BToeXad0jt8P6VpUuf1C_CfPQQWAWbxphwhl-EqrtqhBZObOzlx5MMKKeIvdzXfVN4fff9NIO7l18MidXzTHvWk7m6KP6NjrMeWNYLr3_gjmBhp8d_qhBn2YAnkMi2FG5-6Ij9JM/s1600/debt.png" /></a></div>
This section considers the settings I have configured for my project. In this case the debt has increased from 7.75% to 7.93% due to the increase of number of issues in the solution. It also determines that the time needed to reach band "A" is of 3 hours and 32 min. The total amount of days needed to fix <b>all the issues</b> is the Debt (1 day and 1 hour).<br />
<br />
To get values closer to reality, you have to configure your project to specify how long it will take you or any member of your team to fix an issue (most of the times I just specify half a day per issue as a rule). Here you can see the settings I have specified in my solutions as a rule of thumb and that you can consider in your projects:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4KjhFKfGAFmg0WNn_EI3LQph9x3EJxD6S4d9eh6euK75vtEAtPFFOJh6i6F5MafW0ckXMWtl4KtCWM2ER-2tOJywjxukjcEglZ_pFMCcc-Ng6dOhOTHmJxa2RMe7YU8kGEwjN8xjyADc/s1600/settings.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="953" data-original-width="1358" height="449" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4KjhFKfGAFmg0WNn_EI3LQph9x3EJxD6S4d9eh6euK75vtEAtPFFOJh6i6F5MafW0ckXMWtl4KtCWM2ER-2tOJywjxukjcEglZ_pFMCcc-Ng6dOhOTHmJxa2RMe7YU8kGEwjN8xjyADc/s640/settings.png" width="640" /></a></div>
<br />
These settings use the following considerations:<br />
<br />
<ul>
<li>Your team will mostly code 6 hour a day. The rest of the time is spent with meetings, emails, research, etc.</li>
<li>The estimated effort to fix one issue is of 4 hours. That's the minimum I would give as average. There are issues that are fixed in 5 min and there are others that might take quite a bit of time. Don't forget that this time also includes filling the ticket details in your scrum environment and documentation, etc.</li>
<li>Then depending on the severity of the issue there is a threshold specified too as you can see in the figure above.</li>
</ul>
<br />
Another aspect to consider to get a proper estimation is also the <b>code coverage</b>. If you configure the coverage correctly in your solution then NDepend can get that data and use it to get a more comprehensive estimation.<br />
<br />
To configure code coverage for NDepend you can follow my steps below:<br />
<h3>
<b><br /></b></h3>
<h3>
<b>Configuring JetBrains DotCover.</b></h3>
Once you've run your initial analysis, NDepend will also ask you to configure Code Coverage to get more information about your project and some additional metrics.<br />
<br />
Go to <b>NDepend project coverage settings</b> under the <b>Analysis </b>tab and in there you'll have to select the <b>XML </b>file generated by <b><a href="https://www.jetbrains.com/dotcover/">DotCover</a></b>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGxjlT4ynTIPbzyUvxuXcqdDaFzOd-x2Ftlm8oNcpXTPrEcxhPEUFziTc4Pb6sMx5IYeZ6WwDYXm8-g4r81gvHOtoPgxwORO-pso9zT8RdiN0pd3rTOJjUNTw7z3BjqESr6H7TTCAIqGA/s1600/dotcover.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="683" data-original-width="966" height="451" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGxjlT4ynTIPbzyUvxuXcqdDaFzOd-x2Ftlm8oNcpXTPrEcxhPEUFziTc4Pb6sMx5IYeZ6WwDYXm8-g4r81gvHOtoPgxwORO-pso9zT8RdiN0pd3rTOJjUNTw7z3BjqESr6H7TTCAIqGA/s640/dotcover.png" width="640" /></a></div>
<br />
If you run your tests with <b>ReSharper </b>you can select the coverage option and then in that menu go to the export button and select "<b>Export to XML for NDepend</b>". Leave this file in a known folder so you can automate this easily later on. The goal here is to configure everything manually but then you will have to do the work around so you can trigger all this with your build agent and get the report at the end of the run.<br />
<br />
Chose the exported file and run again your analysis:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfMeY2tKX_hxX_0THCRWqrEpdwkKBaJkrcUQDLLJH1gRVefz0jCI7pugjlHp8hvp_41h12c_icXHX9rc-MwUbC-C8Hv-71pSItD4oB1cQnzzqLeo-1op1o4brcjRjw2cwvNoV1C9-R2as/s1600/file.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="698" data-original-width="1289" height="346" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfMeY2tKX_hxX_0THCRWqrEpdwkKBaJkrcUQDLLJH1gRVefz0jCI7pugjlHp8hvp_41h12c_icXHX9rc-MwUbC-C8Hv-71pSItD4oB1cQnzzqLeo-1op1o4brcjRjw2cwvNoV1C9-R2as/s640/file.png" width="640" /></a></div>
<br />
Now with all these details if you run NDepend you should get something like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgR8hyphenhyphenj3XReKXEZWVvKTzO4fLSGTHxFFT2rGQQ5uLwKeFMwzTAWSxmbQZqGG9fDRcqKHt9eugEdPPauaxCRB1pXNDqJzdUzKB0VL5KUXig4-N0QmCEc1BC-kBqveDNtjZocgXNQPHnQ1js/s1600/result.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="964" data-original-width="1285" height="478" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgR8hyphenhyphenj3XReKXEZWVvKTzO4fLSGTHxFFT2rGQQ5uLwKeFMwzTAWSxmbQZqGG9fDRcqKHt9eugEdPPauaxCRB1pXNDqJzdUzKB0VL5KUXig4-N0QmCEc1BC-kBqveDNtjZocgXNQPHnQ1js/s640/result.png" width="640" /></a></div>
<br />
Now you can see proper debt and the coverage. This is a little project that I'm currently working on and that it really works to demonstrate how good NDepend is in this case. If you don't know what one of the terms means, you can just click on it and you'll be redirected to the panel with all the details about that specific metric and its description.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh96qn12olD0qSLKdTE80R3Ji4oUlIm-5JRDNJyKY5JXNKVN_d4-ynA9cdZBAXDOjbpNFNLysEte62Ss8fwYbK4D2RRki1tJNVbdgICrsus2HRCcKWodm8s28GUMptjlaYpMpS_78a0hbM/s1600/description.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="337" data-original-width="1535" height="140" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh96qn12olD0qSLKdTE80R3Ji4oUlIm-5JRDNJyKY5JXNKVN_d4-ynA9cdZBAXDOjbpNFNLysEte62Ss8fwYbK4D2RRki1tJNVbdgICrsus2HRCcKWodm8s28GUMptjlaYpMpS_78a0hbM/s640/description.png" width="640" /></a></div>
<br />
The following three additional panels help shaping the technical debt information: Quality Gates, Rules and Issues. Below you'll find a quick introduction on each section and its relevance.<br />
<h2>
<b><span style="font-size: large;"><br /></span></b></h2>
<h2>
<b><span style="font-size: large;">Quality Gates</span></b></h2>
<b>Quality gates</b> are based on <i>Rules</i>, <i>Issues </i>and <i>Coverage</i>. Basically this section determines certain parameters that your project should match in order to pass "quality". So for example: your project should contain a % of code coverage, your project should not contain Blocker or Critical issues, etc.<br />
<br />
Here are some of these gates used for your reference:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRtHN7_BlRrVZfBYsmbxuan3YFJeP4LAnPulRpYW-QOIrs9iazzBU8J41Ps8drdIOR4K9ezDIfMfLwt6s1JdakOZOU4pO9NKX1KJbLC6vMm7i_Y5kHTXZnj6RqRRGXWqj_bOgfBhr28i0/s1600/gates.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="182" data-original-width="621" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRtHN7_BlRrVZfBYsmbxuan3YFJeP4LAnPulRpYW-QOIrs9iazzBU8J41Ps8drdIOR4K9ezDIfMfLwt6s1JdakOZOU4pO9NKX1KJbLC6vMm7i_Y5kHTXZnj6RqRRGXWqj_bOgfBhr28i0/s1600/gates.png" /></a></div>
<h2>
<b><span style="font-size: large;"><br /></span></b></h2>
<h2>
<b><span style="font-size: large;">Rules</span></b></h2>
Rules are defined as Project Rules and they check for violations in your code. This is like the rules defined by FXCop and that provide real arguments as to why your code is breaking a rule or that it needs to be better. Once you've gone through several iterations of fixing these, then your code will get cleaner and better (I promise you!). And most of all, you will understand the reason behind the rule!.<br />
<br />
Here are some of these rules:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhM_eQ9c7sAGZ3wu89jL35IlKRF7WteGD8nAR-V5O8mcSdTCO1ThEliFbIYEDZ79vi03dhyphenhyphenld-vf74DDVAQo0Sx1UjKU80oupA0J1gf4bApwbsVuBxc_gALHXZKOpz94IzS6NIAXUY3guc/s1600/rules.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="472" data-original-width="1131" height="267" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhM_eQ9c7sAGZ3wu89jL35IlKRF7WteGD8nAR-V5O8mcSdTCO1ThEliFbIYEDZ79vi03dhyphenhyphenld-vf74DDVAQo0Sx1UjKU80oupA0J1gf4bApwbsVuBxc_gALHXZKOpz94IzS6NIAXUY3guc/s640/rules.png" width="640" /></a></div>
<br />
If you think that one of these rules does not apply to your project, you can just uncheck it and the framework will take of it so you don't have to worry about it anymore.<br />
<h2>
<b><span style="font-size: large;"><br /></span></b></h2>
<h2>
<b><span style="font-size: large;">Issues</span></b></h2>
The number of issues are just a way of grouping the rules so you can determine which ones are the important ones to fix. So you can violate few rules but then these rules are categorised between blocker and low. So even though the project is violating 18 rules, 1 of these rules is just Low. This gives you an understanding of what's important to fix and what can wait.<br />
<br />
Then each issue has a clear definition of the time that could take to fix:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7iWn_N5kHxk6rhUMMjXDepmFX2f-DqTZ8dQvGaiDMc0D800NZAEKo0WfuK5a7zpTe2w3mqLr_TwVweEKb6cP1NKtSpht6yfZokqsbiIz8YpDtMa6S-jKaTPN0pKjz-5i90cUDrI7kOk4/s1600/issues.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="196" data-original-width="727" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7iWn_N5kHxk6rhUMMjXDepmFX2f-DqTZ8dQvGaiDMc0D800NZAEKo0WfuK5a7zpTe2w3mqLr_TwVweEKb6cP1NKtSpht6yfZokqsbiIz8YpDtMa6S-jKaTPN0pKjz-5i90cUDrI7kOk4/s640/issues.png" width="640" /></a></div>
<br />
<b><span style="font-size: large;"><br /></span></b>
<br />
<h2>
<b><span style="font-size: large;">Conclusion</span></b></h2>
To conclude, writing quality code is one of my main concerns nowadays. It's really easy to write code and also code that works but the difference between code that works and excellent code is this: <b>quality </b>and <b>NDepend </b>has the solution for <u>you</u>.<br />
<br />
I have been fiddling with tools like FXCop and NDepend for a while now and I must say that NDepend is a must have in my toolkit belt. Really easy to use and with just one click you can have real arguments on the issues that need to be fixed in your solution and how long the team should take to fix them.</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-78254593471295265292017-05-23T00:06:00.000+02:002017-05-23T00:06:18.431+02:00Pushing Messages from Server to Client Using SignalR2 and MVC5<div style="text-align: justify;">
One of the biggest disadvantages of any web client is that they are <b>stateless</b>. They don't know if anything happens in the server side unless they request the information again and again. In this article <u>you will learn a very useful way of pushing updates from the server to the client using SignalR</u>. The idea behind this concept is very simple; I need to find a way <b>to inform the user</b> that there will be some maintenance occurring shortly on the site and that I need them to close the browser to avoid any data loss. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
This is a very simple and elegant way using the <b><a href="https://www.asp.net/signalr">ASP.NET library SignalR</a></b>. This amazing library is designed to use the existing web transport layer (<a href="https://www.tutorialspoint.com/html5/html5_websocket.htm">HTML5 Websockets</a> and other technologies for old browsers) and it's capable of pushing the data to a wide array of clients like web pages, windows apps, mobile apps, etc. It's extremely easy to use, real-time and it allows Developers to focus on real problem leveraging the communication issues to <b>SignalR</b>.<br />
<br /></div>
<h2 style="text-align: justify;">
<b><span style="font-size: x-large;">Overview</span></b></h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi74Jn4kl7r2BofcF0OT2e9jLkZ0gqMuo9OlWvZvv4fCQG-x3tRH6uOn8WP7oVg-A0rA6bBoWhWYEz1gd53J5X6ORuB05S10hOguvM0qzRbDlW1UYRCWvqFkdQvqAWrhp1hqlaYVndZ-Ck/s1600/signalr.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi74Jn4kl7r2BofcF0OT2e9jLkZ0gqMuo9OlWvZvv4fCQG-x3tRH6uOn8WP7oVg-A0rA6bBoWhWYEz1gd53J5X6ORuB05S10hOguvM0qzRbDlW1UYRCWvqFkdQvqAWrhp1hqlaYVndZ-Ck/s1600/signalr.png" /></a></div>
<div style="text-align: justify;">
The image above shows the idea behind the implementation of the SignalR ecosystem. We need to be able to push a notification from a client and that this message gets broadcasted to every single client that's listening to the SignalR Hub.<br />
<br />
<h2>
<b><span style="font-size: x-large;">Install SignalR</span></b></h2>
In order to use this functionality, first we need to install the <b>SignalR </b>(<b>v2.2.1</b>) library via NuGet package:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9f9qh1FxvAUhQIV8zHb_ari_g4OnLPc2r-UgtG29Mfzl0Uz4vi1Nfs8lt6hNctqzCAqfkaksguWeAJ0IQGx2ut0GFCH0tBRBFnctby-8XuhlHqTDUiV1dcdoVLE9bukofDf_MruMuGC8/s1600/nuget.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="297" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9f9qh1FxvAUhQIV8zHb_ari_g4OnLPc2r-UgtG29Mfzl0Uz4vi1Nfs8lt6hNctqzCAqfkaksguWeAJ0IQGx2ut0GFCH0tBRBFnctby-8XuhlHqTDUiV1dcdoVLE9bukofDf_MruMuGC8/s640/nuget.png" width="640" /></a></div>
<br />
Create a folder called "<b>Hub</b>" in your main <i>MVC </i>solution and then add a new <b>SignalR Hub class</b> called <b>NotificationHub.cs</b> as show below:<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNH7Vgob0sq8SvUT80Mlqwy2ue8s5YiVdCE80I68iw0ttbPtWxvujYV85kWocjhl33fv-fe11cz3rn5PLBZ7jNtgpaY4hCLdWnS_rcZEtAaH2Rhm530SRRayde0tiPz3hwERCVsRqF6QQ/s1600/notif.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="444" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNH7Vgob0sq8SvUT80Mlqwy2ue8s5YiVdCE80I68iw0ttbPtWxvujYV85kWocjhl33fv-fe11cz3rn5PLBZ7jNtgpaY4hCLdWnS_rcZEtAaH2Rhm530SRRayde0tiPz3hwERCVsRqF6QQ/s640/notif.png" width="640" /></a></div>
<br />
This will create a class that inherits from the <b>Hub </b>base class.<br />
<br />
<h2>
<b><span style="font-size: x-large;">Creating the Notification Hub</span></b></h2>
Copy the following template to generate the notification Hub. This Hub needs a method "<b>Broadcast Message to Clients</b>" which accepts a <b>message </b>as a string a <b>user </b>as a string and that every client <u>will receive</u>. It uses the <b><a href="https://docs.microsoft.com/en-us/aspnet/signalr/overview/guide-to-the-api/hubs-api-guide-net-client">Clients.All</a></b> property to access all of the clients that are currently connected to the server (hub). This function is just a client side callback function that we will call from the client JavaScript side.<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/e03d40f5991acfc6319ccab0f050483a.js"></script>
Next step is to create your Startup class where you will be able to enable SignalR. Under <b>App_Start </b>folder, create a new class called <b>Startup,cs</b> and add the following code:<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/9edafe356e3f4c5d5c009c547f3bd8ad.js"></script>
This will allow you to <b>map </b>the available hubs using the <a href="https://docs.microsoft.com/en-us/aspnet/aspnet/overview/owin-and-katana/owin-startup-class-detection">Owin startup</a>. Now that the Hub is ready, we need to focus on the client that will display the message received by it.<br />
<br />
<h2>
<b><span style="font-size: x-large;">Showing the notification on the client</span></b></h2>
Now that the server side code is done, we need to be able to display the notification received by it on the client side. To do this we just need to add the relevant script references and the following code to the <b>_layout.cshtml</b> page:<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/2b1bf1550cca568a5ffd33924ca099d8.js"></script>
This page contains the jQuery and SignalR scripts, the SignalR Hub and the proxy hub object using the "<span style="color: blue;">var notificationHub = $.connection.notificationHub;</span>" command. Notice that notificationHub starts with lower case. <span style="color: red;">This is actually very important! because if you don't write it in lower case the reference will not work!</span>.<br />
<br />
The code works in the following way. When the client connects to the hub, the message "connected to the notification hub" should be visible in your browser console and when a new message is received, the div <b>#notificaiton</b> should empty itself and populate itself with the message received. This div sits on a separate page:<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/11cdcaf02ba89c80c41c5e3bca7a0721.js"></script>
This is the aspect of the page without any notification:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzcz1HW14xV1tIheI4HIuiThlbOObHNzT3qKM5VV3tj_9AWoOdgpEpK7uCL-zSMqQ6dSUOFKVPzVkSWSrD8tpvzsN6u4kGlW2ZywUel1PvhaJ3fXBY9Y5o-PhZjtSrGOK_rgkTqrqAJeI/s1600/notifwindow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="212" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzcz1HW14xV1tIheI4HIuiThlbOObHNzT3qKM5VV3tj_9AWoOdgpEpK7uCL-zSMqQ6dSUOFKVPzVkSWSrD8tpvzsN6u4kGlW2ZywUel1PvhaJ3fXBY9Y5o-PhZjtSrGOK_rgkTqrqAJeI/s640/notifwindow.png" width="640" /></a></div>
<br />
<h2>
<b><span style="font-size: x-large;">Sending the notification to the client</span></b></h2>
Now the interesting part. To send the notification to the client, we can either create a separate screen on our MVC application, or just create a small utility to send the message separately. In this case I will choose the latter as it looks to me like a cleaner approach. So here is the code of my <b>submit</b> message functionality (WinForms) which allows me to send push notifications to all the clients connected to the Hub with just one click:<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/92a559012084ae5d113b38dda0a2123e.js"></script>
Here is the simple screen:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDaDeKBIc_5BjVg4dSRWElfXlm9xiXUKbhfSIJr9rgVJDEn23OsEc5KqT3Lq2cuUhh4rVSA8LomL6XQiE-7vrsgH1Bq1LQAEzVTpeQwqIhuGvefEUXM-jdZauE2B7sGvy6dtzZUFwR2sM/s1600/form.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDaDeKBIc_5BjVg4dSRWElfXlm9xiXUKbhfSIJr9rgVJDEn23OsEc5KqT3Lq2cuUhh4rVSA8LomL6XQiE-7vrsgH1Bq1LQAEzVTpeQwqIhuGvefEUXM-jdZauE2B7sGvy6dtzZUFwR2sM/s1600/form.png" /></a></div>
<br />
Finally, if you want to see the system in action, see the animation below for reference:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioFsEtR5lc9PoyAdYz9dIg-M7xa-xebV4ifXAzA-mLw_xVG3AclLZRpGWKIKih2B0EU73c0XJF-F8vW79JXh3wsFBVFbRhyphenhyphen0_4XlXt414WjZ7tldpxLbZwJquoe0d5Awtg8Iik-vRGYQY/s1600/notifanimation.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="392" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioFsEtR5lc9PoyAdYz9dIg-M7xa-xebV4ifXAzA-mLw_xVG3AclLZRpGWKIKih2B0EU73c0XJF-F8vW79JXh3wsFBVFbRhyphenhyphen0_4XlXt414WjZ7tldpxLbZwJquoe0d5Awtg8Iik-vRGYQY/s640/notifanimation.gif" width="640" /></a></div>
<br />
With this approach, you will be able to inform all your connected users easily, irrespective of the client technology or platform. The source code of the project is still not available on my <a href="https://github.com/JordiCorbilla">GitHub</a> page, but I will make sure to make it available so you can test it locally and see it by yourselves.<br />
<br />
<div style="text-align: right;">
Jordi.</div>
</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-43886343151985920452017-05-20T11:58:00.003+02:002017-05-20T12:07:03.687+02:00Inside refactoring with Delphi<div style="text-align: justify;">
In this article I will show you <b>two common techniques</b> that I use in my C# projects that they are quite relevant for any other programming language out there. In this case Delphi as I'm sure many developers out there can refer to the same principles. <b>The following code has been tested under Delphi 10.2 Tokyo version</b>.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The first technique is quite used in Functional Programming but it can be related to OOP and it's called <b>Imperative Refactoring</b>. The second technique helps reducing common code and eliminates inconsistencies and it's called <b>Inline Refactoring</b>. See the examples below for guidance.<br />
<br /></div>
<h2 style="text-align: justify;">
<b><span style="font-size: x-large;">Imperative Refactoring</span></b></h2>
<div style="text-align: justify;">
This technique is quite easy to understand and I'm sure you've applied this many times in your projects.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwSdZ7hyphenhyphenNgU3-Bdtx9Bjv1DIK-9HBBo1GDxA4xpc9QuFkwQgh6IsX-G9n7URvBgtRQhECGa03-ZCSFVdNscwBDLqELkZjMNyPGkVtIk2WSpDRrt5prxKpEcr_tQuUmVnOC14FAXt8qDYE/s1600/ref1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="250" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwSdZ7hyphenhyphenNgU3-Bdtx9Bjv1DIK-9HBBo1GDxA4xpc9QuFkwQgh6IsX-G9n7URvBgtRQhECGa03-ZCSFVdNscwBDLqELkZjMNyPGkVtIk2WSpDRrt5prxKpEcr_tQuUmVnOC14FAXt8qDYE/s640/ref1.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
In this case, we have a method or function that has some code that we would like to reuse. The principle says that this code needs to be extracted and placed externally onto another function and then add the call where the previous code was. This technique is very simple and very easy to embrace for code reusability.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Here you can see a typical example:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Before Refactoring:</b></div>
<div style="text-align: justify;">
<script src="https://gist.github.com/JordiCorbilla/527bea8cb59d0710c10d81b89230dffe.js"></script></div>
<div style="text-align: justify;">
As you can see this is a very simple example where I request a web page and then I do some parsing to get the list of urls that are part of the html document. Let's see how to refactor it to make it more reusable.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>After Refactoring:</b></div>
<div style="text-align: justify;">
<script src="https://gist.github.com/JordiCorbilla/cc1a2b3558a3329dc5a4ee51ada7043e.js"></script>
</div>
<div style="text-align: justify;">
Notice that I've extracted the parsing functionality and I've created a parseHTML function that gets the response and parses it and returns the list of urls. Now I can reuse my parsing functionality should I have any other page where this functionality is required. No-brainer here.<br />
<br /></div>
<h2 style="text-align: justify;">
<span style="font-size: x-large;">Inline Refactoring</span></h2>
<div style="text-align: justify;">
This one is a bit different and it relates to the outer code as a reusable code. Imagine that we would like to refactor the inline functionality: In this example, I'm repeating quite a lot the functionality to fetch an item from the internet but I would like to reuse it so I can a) replace the http component at any time without impacting the rest of the code and b) replace the parsing part so it can return any kind of object:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhW_ZjzpKvmQ4Zs8eLQLkU2Q25WHsOX9hmDMTXe-a0VTagP4dXntyxEujHywsvwpIvgbJ5WEKuwDVYKmQ55e_n69TyFphngjhJy3SkIwn9GPv-HpmI6ICjKLnYvV8zfF8cqKYMnPN4xxaw/s1600/ref2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="190" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhW_ZjzpKvmQ4Zs8eLQLkU2Q25WHsOX9hmDMTXe-a0VTagP4dXntyxEujHywsvwpIvgbJ5WEKuwDVYKmQ55e_n69TyFphngjhJy3SkIwn9GPv-HpmI6ICjKLnYvV8zfF8cqKYMnPN4xxaw/s640/ref2.png" width="640" /></a></div>
<br />
The idea behind this refactoring is to be able to reuse the external call also using anonymous methods and generics.<br />
<br />
Here is the after refactoring code:<br />
<br />
<b>After Refactoring:</b><br />
<script src="https://gist.github.com/JordiCorbilla/f92189b0e944b49cf3269bdff31ad501.js"></script>
As you can see the idea is to use anonymous methods and generics heavily to be able to reuse most of the functionality and allow the developer to separate the concerns of downloading the page and parsing it. It also allows you to rebuild the component in a different way e.g. in this case I'm using Indy components to request the page but you might like to use another component. Using this approach everything is quite modular and it gives room for testing. Notice that no functionality has changed here.<br />
<br />
You can find the full source code of this example in my personal repository on <a href="https://github.com/JordiCorbilla/thundax-delphi-personal-repository">Github</a>.<br />
<br />
<div style="text-align: right;">
<b>Jordi.</b></div>
<div style="text-align: right;">
<b>Embarcadero MVP.</b></div>
</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-22946421438009368622017-02-26T19:07:00.001+01:002017-03-09T23:23:03.370+01:00Delphi Firemonkey (FMX) rendering TCanvas for Android<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_Bf-Sq_dTULwpikPAa4TmT39odFzvGDFftgD00OGTYeTy9HTXn-i8nIV9XOLD7Z4Ee5lqMd-TEn0-kh5Nyrb-8eTcp6MSekTRAlCWnE9WwfqbDGlYmvPESseAD2jRV-6xOUmkso_gnVQ/s1600/taskmatch.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_Bf-Sq_dTULwpikPAa4TmT39odFzvGDFftgD00OGTYeTy9HTXn-i8nIV9XOLD7Z4Ee5lqMd-TEn0-kh5Nyrb-8eTcp6MSekTRAlCWnE9WwfqbDGlYmvPESseAD2jRV-6xOUmkso_gnVQ/s400/taskmatch.png" width="233" /></a></div>
<div style="text-align: justify;">
I'm currently working on a new game for Android using <b>Delphi Berlin 10.1</b> and one of the issues I faced so far is regarding canvas rendering. I like painting my own stuff using the canvas (<a href="https://github.com/JordiCorbilla/thundax-delphi-physics-engine">here is proof of it</a>). So imagine how I felt when I deployed the game to my Android device and noticed that nothing was being displayed while the game works perfectly under Windows.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The reason behind this change is that the component <b>TImage </b>renders differently now and you have to paint everything on a TBitmap.canvas component and then assign it to the original TImage component so it gets displayed correctly so it requires few tweaks. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The idea of the game is to find and match one of the 6 images shown with the image displayed below. Once you have identified the exact match you just need to tap to it to go to the next game. The game increases difficulty when you score certain number of points. You have just 10 seconds to complete the task.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The way I've built this is by using 6 <b>TImage</b> components and then I render there a matrix of <b>zeros and ones</b> that get represented into an image so the first image on the left is actually the matrix:</div>
<div style="text-align: justify;">
[1,0,0,1]</div>
<div style="text-align: justify;">
[1,0,1,0]</div>
<div style="text-align: justify;">
[1,0,0,1]</div>
<div style="text-align: justify;">
[0,1,0,0]</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The numbers get randomly generated and I make sure that no duplicates are found during the randomisation.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
If you see the image below, by level 20 and 50 things get complicated:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiorBe4zChGO8s51aVWGxpLOCf9pxz03vYxnScg-RANWtgW8u7NCaF0ORAP9eya9b5EeirBv5gmtlLu8i9TG98ExLqxjDHKeH7_iZN-UzbFVM_XKVipIL52m7d298hpgY5vsGVdKv8FPRs/s1600/levels.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="286" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiorBe4zChGO8s51aVWGxpLOCf9pxz03vYxnScg-RANWtgW8u7NCaF0ORAP9eya9b5EeirBv5gmtlLu8i9TG98ExLqxjDHKeH7_iZN-UzbFVM_XKVipIL52m7d298hpgY5vsGVdKv8FPRs/s640/levels.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
If you look at the <b>source code</b> below, this code tries to paint the main core of the application and it will work perfectly well on <b>Windows </b>but it won't work under <b>Android </b>(you will just get a black screen).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<script src="https://gist.github.com/JordiCorbilla/b4eb68aa9cbb1b30a6f1fff457ef628e.js"></script></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
To overcome this issue, we need to do the rendering a bit different. Now we need to create a custom bitmap, paint in there and then assign it to the original TImage. Here is a different version of the code above that works under Windows and Android and everything gets rendered correctly.<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/2856e00382f3801de1404a0158a0e6fa.js"></script><br />
<br />
Here is the game running on Android (just be aware that it's still under beta testing and that there are still many things to fix):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgy8KjmBR6jaSiEM3Rpah7axOEZ1y2-8jNuWNMWDizsPZFt8WCQvJa3EyRumFDJBdxg1hlTk9dxwVJH5-a7YGlIl0_J3xzg0B0JEvLunDZy7ky-GflViTqOuzVbhcvRGNQhq-xTkvGoQIo/s1600/Animation.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgy8KjmBR6jaSiEM3Rpah7axOEZ1y2-8jNuWNMWDizsPZFt8WCQvJa3EyRumFDJBdxg1hlTk9dxwVJH5-a7YGlIl0_J3xzg0B0JEvLunDZy7ky-GflViTqOuzVbhcvRGNQhq-xTkvGoQIo/s640/Animation.gif" width="357" /></a></div>
<br />
Once finished I'll place it in google play for your amusement and share the source code. Let me know if you have any questions.<br />
<br />
I'm still using the same concepts to my Delphi Physics Engine so I can render correctly my models on Android. I'm rewriting the library so it supports FMX and from there I'll be able to render on Android without problems!.<br />
<br />
To load bespoke fonts for Android I had to use the following article which did the treat:<br />
<br />
<ul>
<li><a href="http://firemonkeyblog.blogspot.co.uk/2014/12/using-custom-fonts-in-android-delphi.html">http://firemonkeyblog.blogspot.co.uk/2014/12/using-custom-fonts-in-android-delphi.html</a></li>
</ul>
<br />
<br />
<div style="text-align: right;">
Jordi<br />
Delphi MVP.</div>
</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com1tag:blogger.com,1999:blog-7022755517551007355.post-34239396280970272622017-01-27T21:02:00.003+01:002017-01-28T19:40:13.788+01:00Firebase Cloud Messaging with Delphi 10.1 Berlin update 2.<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8eLbNJ4TGHgRuEHNJQ1WKWSOKgYFLSL1ogMsOUDDFOnIuQtDzq6vsp53m2qBlITOZ-01OnUr73iPm7ncLlrYfSQ9c2tu6o1nDdgNhqgSY9unXmMnfT83bRLIA0kCuzxnTbaLMQRQTAOQ/s1600/push.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="243" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8eLbNJ4TGHgRuEHNJQ1WKWSOKgYFLSL1ogMsOUDDFOnIuQtDzq6vsp53m2qBlITOZ-01OnUr73iPm7ncLlrYfSQ9c2tu6o1nDdgNhqgSY9unXmMnfT83bRLIA0kCuzxnTbaLMQRQTAOQ/s320/push.png" width="320" /></a></div>
A comprehensive step by step guide, covering everything you need to know to receive push notifications to your <b>Android</b> device using <a href="https://firebase.google.com/docs/cloud-messaging/">Firebase Cloud Messaging</a> and the latest Delphi 10.1 Berlin update 2.<br />
<br />
<span id="goog_1403160535"></span>Push notifications let your application notify a user of new messages or events even when the user is not actively using the application<span id="goog_1403160536"></span> (downstream messaging) (<a href="https://parse.com/tutorials/android-push-notifications">Parse.com</a>). On Android devices, when a device receives a push notification, the application's icon and a message appear in the status bar. When the user taps the notification, they are sent to the application. Notifications can be broadcast to all users, such as for a marketing campaign, or sent to just a subset of users, to give personalised information. To provide this functionality I will rely on <b>Firebase Cloud Messaging</b> which is the new version of <b>GCM</b> (<a href="https://developers.google.com/cloud-messaging/">Google cloud messaging</a>) and Delphi to develop the Android application.<br />
<br />
<h2 style="font-weight: bold;">
1. Create your Firebase project</h2>
<br />
Create your <a href="https://console.firebase.google.com/">Firebase project</a> by visiting the console if you still don't have one and in this project create an Android App.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLqBDwZZ45Sj41-TYoWfZBtgKTKhFLFtuZzOU3zxBnP06tl97zXRRuDPW-3oFXN9dPOf6ZBW-IqKuqnVVr8MwOfgZp3m7cfeX-vojVAMGIKvHCUpnkBfmkpm9oFvdBzuKAxuoz3xaazpU/s1600/newproject.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="280" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLqBDwZZ45Sj41-TYoWfZBtgKTKhFLFtuZzOU3zxBnP06tl97zXRRuDPW-3oFXN9dPOf6ZBW-IqKuqnVVr8MwOfgZp3m7cfeX-vojVAMGIKvHCUpnkBfmkpm9oFvdBzuKAxuoz3xaazpU/s640/newproject.png" width="640" /></a></div>
<br />
I already have one project so I will use this one for my demo. Once in the project, go to <b>Overview -> Add another app -> Android:</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4mZHYt7O8r8d6Kky7xqS6-qizrXUpwG0CVfNaiIDfGAw43onFFzpZ4EZTNX3zRCxyD1NSAlgDcmPw5ojkHGgLNLJQWdBo4-uxuilSt_ItZN35m0sR8nX3T44hmRk4ii0Jwxln0_CmnwY/s1600/addapp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4mZHYt7O8r8d6Kky7xqS6-qizrXUpwG0CVfNaiIDfGAw43onFFzpZ4EZTNX3zRCxyD1NSAlgDcmPw5ojkHGgLNLJQWdBo4-uxuilSt_ItZN35m0sR8nX3T44hmRk4ii0Jwxln0_CmnwY/s640/addapp.png" width="600" /></a></div>
<br />
And give it a sensible name. In my case I called the package <b>com.embarcadero.FirebaseCloudMessaging</b>. This package name is important as it will be referenced later on. Once you click Add App, you will receive a <b>google-services.json</b> file which contain information that we will use later.<br />
<br />
The package name is defined in your Delphi project:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghszlZGQvWrKLlkHW-L4HYxaWyAH0VKJX32RukgjFUz_7NHmJvW7KccsMC-smvBDPaFiLE_ZM0JW7jnUyMn4-N8vkm61wff8K69rfDyjkovdm0QS4_f88lyy8zjtZJ6UkE25fmLluO3FQ/s1600/android.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="188" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghszlZGQvWrKLlkHW-L4HYxaWyAH0VKJX32RukgjFUz_7NHmJvW7KccsMC-smvBDPaFiLE_ZM0JW7jnUyMn4-N8vkm61wff8K69rfDyjkovdm0QS4_f88lyy8zjtZJ6UkE25fmLluO3FQ/s640/android.png" width="640" /></a></div>
<br />
So make sure that everything matches with the name you give to your Firebase application as the manifest file will contain this information.<br />
<br />
<h2 style="font-weight: bold;">
2 Request your FCM Token</h2>
<br />
Now that we have our project configured, we need to request <b>Firebase </b>a unique <b>token</b> for our Android device. You can see the description here as to how to get the <a href="https://firebase.google.com/docs/cloud-messaging/android/client">FCM token</a> via Android Studio, but I will show the necessary steps to get the same value from our Delphi application.<br />
<br />
Basically we are trying to get the same value from <b>FirebaseInstanceId.getInstance().getToken()</b>. We will achieve the same behaviour by using <a href="http://docwiki.embarcadero.com/Libraries/Berlin/en/System.PushNotification.TPushServiceManager">TPushServiceManager</a> which is the unit responsible for handling push notifications.<br />
<br />
The following code snippet tries to request the FCM token via <b>TPushServiceManager</b>:<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/70f3fd93d09e8f43131125c7106c5000.js"></script>
Now, to allow this code to work correctly, we will have to configure few things.<br />
<br />
<b>a) Enter the <u>Sender Id</u>.</b><br />
<br />
In the source code snippet above, I'm mentioning the SENDER ID. This sender id, can be found under <b>Firebase -> Project Settings -> Cloud Messaging</b>:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbN8V5PcCD2u8Z9Qn5EMjfjFdr_7qB5s3Tu2DymgWqSOU4YY5A6ebFQsbCKAvD6GWmWk9CGvOhk_ucJE3w1NF8pLcHlFU2klFowis3pRP52K9PABGl7z6QhIYWffTg5o4R1wibZIZ8Tp4/s1600/sender.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="262" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbN8V5PcCD2u8Z9Qn5EMjfjFdr_7qB5s3Tu2DymgWqSOU4YY5A6ebFQsbCKAvD6GWmWk9CGvOhk_ucJE3w1NF8pLcHlFU2klFowis3pRP52K9PABGl7z6QhIYWffTg5o4R1wibZIZ8Tp4/s640/sender.png" width="640" /></a></div>
<br />
This is the value you have to put here:<br />
<br />
PushService.AppProps[TPushService.TAppPropNames.GCMAppID] := '<b>SENDER ID</b>';<br />
<br />
Knowing that the <b>GCMAppId</b> is actually the <b>Sender Id</b> has been a quite a struggle for some users and you can see my answer on <a href="http://stackoverflow.com/questions/40888708/delphi-10-1-android-how-to-get-firebase-device-token/41797758#41797758">Stack overflow</a>.<br />
<br />
<b>b) Configure the project to receive push notifications.</b><br />
<br />
In the Delphi IDE, go to your <b>project options</b> -> <b>Entitlement List</b> and set the property <b>Receive push notifications</b> to <b>true</b>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0bA-aIbsCSRcsG4w8rqouscwXi6_5guQAfh0GK3hjltM5UfjsrHWzPX5xe-NycWZnro43v-K8HefBscZKB10fBa_u0hyjQ1LcuQ10TI2ayS8WyTan2CAY0UbE7fkrgwdr_Ks2oANiUoE/s1600/projc.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="336" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0bA-aIbsCSRcsG4w8rqouscwXi6_5guQAfh0GK3hjltM5UfjsrHWzPX5xe-NycWZnro43v-K8HefBscZKB10fBa_u0hyjQ1LcuQ10TI2ayS8WyTan2CAY0UbE7fkrgwdr_Ks2oANiUoE/s640/projc.png" width="640" /></a></div>
<br />
<b>c) Configure the <u>AndroidManifest.template.xml</u> file.</b><br />
<br />
Before we try to run the code above, we'll have to configure the <b><a href="http://docwiki.embarcadero.com/RADStudio/Seattle/en/Preparing_an_Android_Application_for_Deployment#Customizing_Your_AndroidManifest.xml_File">manifest</a> </b>file to grant our device permissions to connect to <b>Firebase</b>. If you don't configure the permissions, you might run into an exception like the one below:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwq53qvb8vxpn0rOEcdC3aOwvg9TFXG5LVe_KsC4rqifTF4iJ0Wt-OmcUEfR3KoF3LdXORPxQt4XA3DANO9Kqe9594Acl1bQTFeFBUeNqF_5CPu5TZoDQQMDybadaRkhhLaoPZu3ATtiE/s1600/exception.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwq53qvb8vxpn0rOEcdC3aOwvg9TFXG5LVe_KsC4rqifTF4iJ0Wt-OmcUEfR3KoF3LdXORPxQt4XA3DANO9Kqe9594Acl1bQTFeFBUeNqF_5CPu5TZoDQQMDybadaRkhhLaoPZu3ATtiE/s640/exception.png" width="640" /></a></div>
<br />
<b>Error message:</b> <i><span style="color: red;">EJNIException with message 'java.lang.SecurityException: Not allowed to start service Intent { act=com.google.android.c2dm.intent.REGISTER pkg=com.google.android.gms (has extras) } without permission com.google.android.c2dm.permission.RECEIVE'</span></i>.<br />
<br />
See the code snipped below for reference:<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/5cd7b674c945246ffecff1981b9e5893.js"></script>
The full source code of the solution can be found <a href="https://github.com/JordiCorbilla/BaaSDelphiSamples"><b>here</b></a> for reference where you can find the manifest files.<br />
<br />
<ul>
<li><a href="https://github.com/JordiCorbilla/BaaSDelphiSamples">Baas Delphi Samples</a>.</li>
</ul>
Once everything is configured, we can now test if we can receive the <b>FCM token</b>. Here is a screenshot of my project so you can see that there are two buttons, one to receive the token and the other one to store this token somewhere so the system that sends the notification knows the receiver.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAl9KHkQCTW0MXUGZxxj8nvFhEWqME5FZ6jCxKuzQOzxzImAHpWJG67a5ipcYUZuzf-7w3-oOWYV_AsupxIo8xbVMEmafCruyn18V8tfoiAdHGxRKnqmVp22OEIJbTfEEmIeHAtAxrCCA/s1600/project.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="248" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAl9KHkQCTW0MXUGZxxj8nvFhEWqME5FZ6jCxKuzQOzxzImAHpWJG67a5ipcYUZuzf-7w3-oOWYV_AsupxIo8xbVMEmafCruyn18V8tfoiAdHGxRKnqmVp22OEIJbTfEEmIeHAtAxrCCA/s640/project.png" width="640" /></a></div>
<br />
Let's see the project in action here:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZl5CJMmUDx0xuIhXyOziQ2colEI7AXWEP71Y8Xkc4sWxUhpQteeifLaDz73nXBHigwGzQDBXzaflqSuOlBmAjNlr01ArJ9ermttdjHp6k5wtrfFHUyjFm2IuCxv6jCnUQL8NhUkkQyEY/s1600/projectaction.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="362" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZl5CJMmUDx0xuIhXyOziQ2colEI7AXWEP71Y8Xkc4sWxUhpQteeifLaDz73nXBHigwGzQDBXzaflqSuOlBmAjNlr01ArJ9ermttdjHp6k5wtrfFHUyjFm2IuCxv6jCnUQL8NhUkkQyEY/s640/projectaction.gif" width="640" /></a></div>
<br />
As you can see in the image above, I get the <b>DeviceID </b>and the <b>FCM Token</b>. The one we are interested in is the <b>FCM Token</b>. This token is quite large so it does not appear completely on the screen.<br />
<br />
Now we need to configure what to do when we receive a notification and how this notification is built.<br />
<h2 style="font-weight: bold;">
3 Receive your first FCM Push notification</h2>
<br />
The following code snipped will configure the <b>OnReceiveNotification</b> event and will display a notification using the <a href="http://docwiki.embarcadero.com/Libraries/Seattle/en/System.Notification.TNotificationCenter">TNotificationCenter</a> class.<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/816c6fdddffa839f18231d26c1b1ad69.js"></script>
Notice that the ServiceNotification variable contains a DataKey member which contains a JSON envelope. This object will contain all the information of our push notification. Here you can see how this notification looks like:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEzic6L9QiJuHGYkQZL-cva4DEARlx4lrm5qwkabDyekG9bO-9K2hY7-0kawTDLObpm2vqawhzSf2W37liY7RhoExdoug717BAF-Nae8li2j60RTPC6dH2CmLQx2FslkQPhnvEsgSEs-k/s1600/envelope.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="275" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEzic6L9QiJuHGYkQZL-cva4DEARlx4lrm5qwkabDyekG9bO-9K2hY7-0kawTDLObpm2vqawhzSf2W37liY7RhoExdoug717BAF-Nae8li2j60RTPC6dH2CmLQx2FslkQPhnvEsgSEs-k/s640/envelope.png" width="640" /></a></div>
<br />
Notice that the message is part of the <b>gcm.notification.body</b> property and this one is the one that we will use for our <b>DataObject.GetValue</b> method.<br />
<br />
Let's see the application in action:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0l_g3xJsUSwDhQuYVar-SjfiSRnXFtkVrqD-Y_nkJ5PBQB6cq4tQPl5GzLh7K25UvbmtUGal-v9ktoTKSF8ea7H9DMiUFidwLopWiTRYjDoBV8iXboWxRSgj1BuezWerUnB7Gh_Lzcfc/s1600/pushnotif.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="332" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0l_g3xJsUSwDhQuYVar-SjfiSRnXFtkVrqD-Y_nkJ5PBQB6cq4tQPl5GzLh7K25UvbmtUGal-v9ktoTKSF8ea7H9DMiUFidwLopWiTRYjDoBV8iXboWxRSgj1BuezWerUnB7Gh_Lzcfc/s640/pushnotif.gif" width="640" /></a></div>
<br />
<br />
Here you can see side by side my Android device and <b>Firebase Notification testing</b> system. Once the application is ready to receive notifications, you just need to send the notification using the configured app or the token if you want to target a single device.<br />
<br />
Next step is to store these tokens on the cloud and use your own system to deliver those messages.<br />
<br />
Please, do not hesitate to contact me if you have any further questions.<br />
<br />
<b>Related links:</b><br />
<br />
<ul>
<li><a href="http://stackoverflow.com/questions/35590533/gcm-in-delphi-seattle-without-baas">GCM in Delphi Seattle without BAAS</a>.</li>
<li><a href="http://stackoverflow.com/questions/25205478/issue-with-gcm-push-notification-service-delphi-xe6/27690786">Issue with GCM Push notifications service Delphi XE6</a>.</li>
<li><a href="http://stackoverflow.com/questions/22117317/securityexception-not-allowed-to-start-service-intent-act-com-google-android">SecurityException: not allowed to start service intent</a>.</li>
<li><a href="https://firebase.google.com/docs/notifications/android/console-device#access_the_registration_token">Send a notification to an android device</a>.</li>
<li><a href="https://www.embarcadero.com/rad-in-action/baas">RAD in Action: Build connected multi-device apps using BaaS</a>.</li>
<li><a href="https://www.b4x.com/android/forum/threads/android-push-notification-gcm-framework-and-tutorial.19226/">GCM framework tutorial</a>.</li>
<li><a href="http://stackoverflow.com/questions/38769388/what-do-i-use-as-the-user-token-to-subscribe-users-to-topics-in-firebase">User Token to subscribe topics in Firebase</a>.</li>
<li><a href="http://stackoverflow.com/questions/38041829/using-fcm-token-obtained-via-batchimport-ios">Using FCM token obtained via batchimport (iOS)</a>.</li>
<li><a href="http://stackoverflow.com/questions/37673205/what-is-the-authorization-part-of-the-http-post-request-of-googles-firebase-d">Authorization part of http post request FCM</a>.</li>
</ul>
<br />
<div style="text-align: right;">
Jordi</div>
<div style="text-align: right;">
Embarcadero MVP.</div>
</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com107tag:blogger.com,1999:blog-7022755517551007355.post-16184193888985100932016-12-18T17:24:00.001+01:002016-12-18T17:28:04.650+01:00Creating a publicly accessible REST Web API on a Raspberry Pi3<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJyHvvuv9nGeRuEQgUENDDXsP6SixHF2t7CPuamrdXzCRMueXNGzBeOcvt5C6CVdrGoJPOtkmt1Kx3ML7omHDSy30NvIV8dAo3ypJPlXqLpM-cIngGkcxLVMjmClVX1LZSjSEC1ziEexY/s1600/architecture.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="239" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJyHvvuv9nGeRuEQgUENDDXsP6SixHF2t7CPuamrdXzCRMueXNGzBeOcvt5C6CVdrGoJPOtkmt1Kx3ML7omHDSy30NvIV8dAo3ypJPlXqLpM-cIngGkcxLVMjmClVX1LZSjSEC1ziEexY/s320/architecture.png" width="320" /></a></div>
<div style="text-align: justify;">
One of the easiest ways to expose your Raspberry Pi publicly is by using <a href="https://ngrok.com/">ngrok</a>. Ngrok is a lightweight alternative to VPN that creates <i>introspected tunnels to localhost</i>. If you have been following my articles, you will know that I managed to built a <a href="http://thundaxsoftware.blogspot.co.uk/2016/09/delphi-rest-client-that-talks-to-rest.html">REST Web API</a> to check the status of my <a href="http://thundaxsoftware.blogspot.co.uk/2016/07/creating-raspberry-pi-3-cluster.html">Raspberry Pi Grid</a>. The project works on my local network and I also have an <b>Android</b> app also that checks if the Grid is alive or not. But the point of building all this is that I should be able to check the status of it anywhere just to make sure that the system is up and running. The easiest way I could find without having to forward ports on my router is by just running Ngrok. </div>
<h3 style="text-align: justify;">
<b><span style="font-size: large;"><br /></span></b></h3>
<h3 style="text-align: justify;">
<b><span style="font-size: large;">Why do you need to expose your Raspberry Pi?</span></b></h3>
<div style="text-align: justify;">
<br />
Get the latest version of <b>Ngrok</b> for <b>ARM</b> <b><a href="https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-arm.zip">here</a> </b>and unzip the content of it on your local computer (in my case a Windows 10 machine). Then transfer the unzipped file to your Raspberry Pi using your preferred method (in my case I use <b>WinSCP</b>).</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEjZe34iefvD-s2SABkDLViCd-ypZ4qa-UrcgICJWCBzN-osOYQKn8R2m5tE_cmhvdWgeXv8FBD33v9TYrdma-53xssNtpdYh0xY1kjRCAjazqLIZwB3vNgVIRR5EwSUdXAAmi-4Iwa1g/s1600/transfer.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="355" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEjZe34iefvD-s2SABkDLViCd-ypZ4qa-UrcgICJWCBzN-osOYQKn8R2m5tE_cmhvdWgeXv8FBD33v9TYrdma-53xssNtpdYh0xY1kjRCAjazqLIZwB3vNgVIRR5EwSUdXAAmi-4Iwa1g/s640/transfer.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Now that the file is in the Raspberry Pi, we need to give it running permissions using the chmod command (<b>I assume here that you know how to SSH to your RPi using putty</b>):</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoJeHs9snfUw49M_g1CUleKK-l3TlwqTKbxXY4tHA7AfBz4LkGAgJOxYP7DeFbg9RraEYPIsgXI35E0YKYcvKpyu6qA7SBmXkyON423C44Kz0AaXetDVhUdKBtgWvw5qrn5fv-WGpyp8Y/s1600/command.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoJeHs9snfUw49M_g1CUleKK-l3TlwqTKbxXY4tHA7AfBz4LkGAgJOxYP7DeFbg9RraEYPIsgXI35E0YKYcvKpyu6qA7SBmXkyON423C44Kz0AaXetDVhUdKBtgWvw5qrn5fv-WGpyp8Y/s640/command.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Now we just need to run the ngrok command: <b>./ngrok http 3000 </b>and all our <b>http </b>traffic on port <b>3000 </b>will be automatically forwarded:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgalFHVT9pzWtVyZst4IMgdGZ9yoIQ6hSFfYqcX0fgNwWpXhPWAP3DFnaMp-PcXmg-dsGPxLlV4NS5s7F_3K9m4vCPR0WTTpk9Z07fqVgnLVpz0lrcafardQJfbOUGmTY59uE32qrpJbX8/s1600/commandn.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="124" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgalFHVT9pzWtVyZst4IMgdGZ9yoIQ6hSFfYqcX0fgNwWpXhPWAP3DFnaMp-PcXmg-dsGPxLlV4NS5s7F_3K9m4vCPR0WTTpk9Z07fqVgnLVpz0lrcafardQJfbOUGmTY59uE32qrpJbX8/s640/commandn.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Notice that now, Ngrok is giving you a public URL that points to your local Raspberry Pi. This means that Ngrok is providing you with a publicly accessible IP address and domain name to your localhost:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJImwKn1tUDqDpwRJqBc0csDmPFXGuf1jZSaVrEoOfkVoO3YaFnuzes7n6KvCx3RTEIbCQUWwzv_Tf6Mtf2oOrKo-5igLXrq6PvqKt2uIi48ORAMbVu8tstVov1gbJNwGUorW3MiGVZIU/s1600/ngrok.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="450" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJImwKn1tUDqDpwRJqBc0csDmPFXGuf1jZSaVrEoOfkVoO3YaFnuzes7n6KvCx3RTEIbCQUWwzv_Tf6Mtf2oOrKo-5igLXrq6PvqKt2uIi48ORAMbVu8tstVov1gbJNwGUorW3MiGVZIU/s640/ngrok.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Now that Ngrok is running, I need to run my Web API Service on that port 3000. To do that, I will just launch another session to my Raspberry Pi and start my <b>node.js</b>:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjej-3ODHK8lJCexJ8d8IRbcSR4K31gtT41PkUyrlUiq8yjq1c-iybWtEgHaWOcZEOD8d73N75n-CJUtMRxrqL_LtXF9_Y4-hGyETEXr48sx_xi0sqLeCnI0ZBWh-hFJVu-qRaPawLm8hA/s1600/runweb.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="202" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjej-3ODHK8lJCexJ8d8IRbcSR4K31gtT41PkUyrlUiq8yjq1c-iybWtEgHaWOcZEOD8d73N75n-CJUtMRxrqL_LtXF9_Y4-hGyETEXr48sx_xi0sqLeCnI0ZBWh-hFJVu-qRaPawLm8hA/s640/runweb.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Notice that now I can browse that URL (<a href="http://61184bba.ngrok.io/">http:/61184bba.ngrok.io/</a>) and check the status of my Raspberri Pis over the internet:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl7nFor53LwVxUAQq-8pOFlCfQnfwu1ikHQY9r05yAGAxqIvSpW1DpfzD2dCvlfnIB6R5p1uV_iczQV4F7GyBVh4RAnNLsOfP60JKMDef0284Tz5wOBumMHoPmhiHT-_Ud9BthuqLIw34/s1600/urlexplore.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl7nFor53LwVxUAQq-8pOFlCfQnfwu1ikHQY9r05yAGAxqIvSpW1DpfzD2dCvlfnIB6R5p1uV_iczQV4F7GyBVh4RAnNLsOfP60JKMDef0284Tz5wOBumMHoPmhiHT-_Ud9BthuqLIw34/s1600/urlexplore.png" /></a></div>
<div style="text-align: justify;">
Notice that every time your stop/start ngrok, <b><u>a new URL</u></b> is being generated. As you are exposing your data over the internet you need to make sure that this URL is just known by yourself and that you add extra security on your API if required (Now the Raspberry Pi is able to take requests from anyone on the web!).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Using this approach you can publish your applications easily so other can see them while developing them and you can even set up a home server to share data, etc. Possibilities are endless here.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
If you check my mobile application (created with my <b>Delphi 10.1 Berlin update 2</b>) you will see that I just need to point the <b>RestClient </b>component to the exposed URL and voila! we'll be able to access that service over the internet:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOj9dXwMyKaKJrw1aaMWJZ8AusswEJbq4XZ9_XRf_4liP_zJykEq8EFURjknvgOctifkx60R6Tl0nN21_y7qc5injboE32U-ogbksRgqJRPYf-7yXLyWXCRml_2UES27UZxqZqJWctNzc/s1600/delp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOj9dXwMyKaKJrw1aaMWJZ8AusswEJbq4XZ9_XRf_4liP_zJykEq8EFURjknvgOctifkx60R6Tl0nN21_y7qc5injboE32U-ogbksRgqJRPYf-7yXLyWXCRml_2UES27UZxqZqJWctNzc/s640/delp.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
All the <b>source code</b> is available <a href="https://github.com/JordiCorbilla/Delphi-RaspberryPi">here</a>:</div>
<div style="text-align: justify;">
</div>
<ul>
<li>Mobile application (Delphi 10.1 Berlin update 2).</li>
<li>Node.js server (Web Api).</li>
</ul>
<span style="text-align: justify;">Here is the application in action:</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx-o9AKAzyV4AZqwyynWp5Jv7mBvsouYnXk64bD-O2Gdki88wbKxbbPMFk0Wo9GlohgOuQhvNm99cOCy0KTTQR003qOwXxHBTu2YqHcoZAHfSkxDxGp_zDgXx3xwuAGXTH7JDxUSLv3r0/s1600/Animation3.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx-o9AKAzyV4AZqwyynWp5Jv7mBvsouYnXk64bD-O2Gdki88wbKxbbPMFk0Wo9GlohgOuQhvNm99cOCy0KTTQR003qOwXxHBTu2YqHcoZAHfSkxDxGp_zDgXx3xwuAGXTH7JDxUSLv3r0/s640/Animation3.gif" width="355" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Let me know if you have any questions.</div>
<div style="text-align: justify;">
Jordi</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com0tag:blogger.com,1999:blog-7022755517551007355.post-21759055460364021822016-12-11T21:22:00.003+01:002016-12-17T22:33:02.983+01:00“Could not load SSL library” Error message on Android 6 Marshmallow with TidHTTP<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxLSASsK_dpacIw0B5IQAPbR5t4T3S5dsl8FOnmk6oFysZ0zUz75WAZecVPsx6B0bhlTsXjYJ6txvqWOGwItpYxPdL-HpvCdHhSbE3Ec0Wls7nBtj0T600BB8puqaoFKX34U-Lb4O2Gdw/s1600/Marshmallow-0.33392800-1450263759--thumb.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="192" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxLSASsK_dpacIw0B5IQAPbR5t4T3S5dsl8FOnmk6oFysZ0zUz75WAZecVPsx6B0bhlTsXjYJ6txvqWOGwItpYxPdL-HpvCdHhSbE3Ec0Wls7nBtj0T600BB8puqaoFKX34U-Lb4O2Gdw/s320/Marshmallow-0.33392800-1450263759--thumb.jpg" width="320" /></a></div>
<div style="text-align: justify;">
I have recently upgraded from <b>Android 4.4 KitKat</b> to <b>Android 6 Marshmallow</b> and to my surprise I found out that the some of my applications stopped working. The main issue is related to HTTPS communications using <a href="http://www.indyproject.org/index.en.aspx">Indy components</a>. As you might know Indy does not implement SSL natively. What the component does is to implement a flexible IOHandler architecture that allows for any SSL implementation to be plugged into Indy. Indy itself implements its own IOHandler class that is based on <a href="https://www.openssl.org/">OpenSSL</a>. </div>
<div style="text-align: justify;">
The problem is that for Android 6, <a href="https://www.imperialviolet.org/2015/10/17/boringssl.html">Google decided not to support OpenSSL</a> anymore. Google replaced OpenSLL with a custom fork called <a href="https://boringssl.googlesource.com/boringssl/">BoringSSL</a> to <a href="http://arstechnica.com/security/2014/06/google-unveils-independent-fork-of-openssl-called-boringssl/">meet it's needs</a>. BoringSSL includes additional changes and patches to the OpenSSL API and it's not backwards-compatible with OpenSSL. Android 6.0 will use the pre-loaded BoringSSL binaries when any application tries to load the OpenSSL library and it will throw an error (the reason why is because BoringSSL and OpenSSL share the same file names).</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
If you try to debug your Android application, you will see the following message when trying to use the TIdSSLIOHandlerSocketOpenSSL handler:</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxk1Vf8y39tJ9Wdjh5QYui737fjg6UwA-Hswy0gdL_e83kXsdQsYdGD6-sG21petF8l3oIHCInosOSFgMXRSSA0FVg-Y4jK0HpnxhOaHTyDmg10xzCZPirgI0yFSDd5-hjFCEQELBrfQo/s1600/error.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxk1Vf8y39tJ9Wdjh5QYui737fjg6UwA-Hswy0gdL_e83kXsdQsYdGD6-sG21petF8l3oIHCInosOSFgMXRSSA0FVg-Y4jK0HpnxhOaHTyDmg10xzCZPirgI0yFSDd5-hjFCEQELBrfQo/s1600/error.png" /></a></div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: justify;">
<b>How to fix this error?</b></h3>
<div style="text-align: justify;">
I found few comments in different forums and in the end the solution was not clear at all and I had to spend few hours fixing the issue myself. The best way to solve this is by deploying a recompiled version of the OpenSSL libraries with your Android application and then tell Indy to use those files. Indy operates at the NDK level so to make Indy avoid BoringSSL you will have to do that.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Get a precompiled version of <b>OpenSSL</b> here:</div>
<div style="text-align: justify;">
</div>
<ul>
<li><a href="https://forums.embarcadero.com/thread.jspa?messageID=826129&tstart=0">AndroidOpenssl1.0.2h.zip</a></li>
</ul>
<br />
<div style="text-align: justify;">
Place these files in your project and reference them in the deployment section. Go to <b>Project -> Deployment</b> and add the two files that are inside of that zip file and specify the destination folder in your Android device as <b>.\assets\internal</b>.</div>
<div style="text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqcG0a88w-y8chZqQGngW-L1x2pfjJNdwEekznwZiXnS0ofOsKSqfVMHHw3ZEsLUVlV8kcFusUAsaiiGi955K6wv3885sBps_sfRqbgB-J413ezv112PuI8DG-rp0Htut5qZtIoZvHt5A/s1600/deployment.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="252" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqcG0a88w-y8chZqQGngW-L1x2pfjJNdwEekznwZiXnS0ofOsKSqfVMHHw3ZEsLUVlV8kcFusUAsaiiGi955K6wv3885sBps_sfRqbgB-J413ezv112PuI8DG-rp0Htut5qZtIoZvHt5A/s640/deployment.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Now the last thing to do is to tell Indy where to look for these files. To do this, you will have to call the IdOpenSSLSetLibPath method with the TPath.GetDocumentsPath during startup (FormCreate for example). Here is an example:</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<script src="https://gist.github.com/JordiCorbilla/8bc40bad03e1d539b0bb13e6832b3066.js"></script></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Once completed, your applications should be able to use the correct version of Indy. You can review my solution and test it here:<br />
<br />
<ul>
<li><a href="https://github.com/JordiCorbilla/BaaSDelphiSamples">Baas Delphi Examples</a>.</li>
</ul>
</div>
<div style="text-align: justify;">
If you need more help you can check the following threads:<br />
<br />
<ul>
<li><a href="http://blog.marcocantu.com/blog/using_ssl_delphi_ios.html">Using OpenSSL in Indy iOS</a>.</li>
<li><a href="http://stackoverflow.com/questions/37105600/error-could-not-load-ssl-library-on-android-with-tidhttp">Error could not load SSL Library</a>.</li>
<li><a href="http://stackoverflow.com/questions/16367842/indy-10-and-openssl">Indy 10 and OpenSSL</a>.</li>
<li><a href="https://github.com/emileb/OpenSSL-for-Android-Prebuilt/tree/master/openssl-1.0.2">OpenSSL for Android Pre-built</a>.</li>
<li><a href="https://forums.embarcadero.com/thread.jspa?threadID=211089">OpenSSL latest android binaries</a>.</li>
</ul>
</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com5tag:blogger.com,1999:blog-7022755517551007355.post-35924872433219936512016-11-27T21:10:00.000+01:002016-11-27T21:10:40.508+01:00Firebase file streaming with Delphi.<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXHt9sgbeQT4HFZAM-yMknBwL5YqMaKTM2r4z_CGfY3INgO0yaYYC3dYWYt2XHnV8mHdjIWmma-0s0NpRNdsDxhj5u4FpVBbathvvudmoFbRPaYyvqDgeIU7huansL_DwzBxh-eZKzXow/s1600/firebasediagram.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXHt9sgbeQT4HFZAM-yMknBwL5YqMaKTM2r4z_CGfY3INgO0yaYYC3dYWYt2XHnV8mHdjIWmma-0s0NpRNdsDxhj5u4FpVBbathvvudmoFbRPaYyvqDgeIU7huansL_DwzBxh-eZKzXow/s1600/firebasediagram.png" /></a></div>
<div style="text-align: justify;">
I'm trying different <a href="https://en.wikipedia.org/wiki/Mobile_backend_as_a_service">BaaS</a> alternatives as Parse.com <a href="http://blog.parse.com/announcements/moving-on/">is shutting down</a> by the end of the year. Luckily for us, we have a really cool alternative: <a href="https://firebase.google.com/">Firebase</a>. Firebase is a unified app platform for Android, iOS and mobile web development. It features <b>RealTime Database</b>, <b>user authenticator</b> and <b>hosting</b> to make application development easier. It provides web app and mobile app developers with a way to link their applications to back-end cloud storage. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
In this article I will show you how I interact with the Firebase RealTime <b>Database</b> via the REST API to stream files in JSON format using the latest <b>Delphi 10.1 Berlin update 2</b>.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The following example is just a <b>POC </b>and it will allow you to upload a <b>pdf </b>file via the desktop application, marshall its content into <b>JSON</b> format and then send it via POST to the <a href="https://firebase.google.com/docs/reference/rest/database/">Firebase Database URL Rest end-point</a>. Then the files are accessed with the (Android) mobile version of it where files are unmarshalled via HTTP Response into <b>JSON </b>format and then transformed back into <b>pdf </b>format. The following diagram explains the process in detail:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit1_e62yzbCgzpYBkV8pCN9huI4l3ZYgSdjE5jdbob25q738FrSFL9AQahEOigJn_i6C1iNe1ows_0L2O5wW1FgApupQ9h_bFOjxzeHjiKNDajzF6rqkmuvobaxvS_Kk70hWjixmeHgwA/s1600/jsonenvelope.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="460" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit1_e62yzbCgzpYBkV8pCN9huI4l3ZYgSdjE5jdbob25q738FrSFL9AQahEOigJn_i6C1iNe1ows_0L2O5wW1FgApupQ9h_bFOjxzeHjiKNDajzF6rqkmuvobaxvS_Kk70hWjixmeHgwA/s640/jsonenvelope.png" width="640" /></a></div>
<div style="text-align: justify;">
The following code shows how to stream the file into JSON format using the <a href="http://docwiki.embarcadero.com/Libraries/Seattle/en/Data.DBXJSONCommon.TDBXJSONTools.StreamToJSON">TDBXJSONTools.StreamToJSON</a> method:</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/JordiCorbilla/a48362022ea36f9290d693ce705bcba3.js"></script></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
And to translate it back to a file stream:<br />
<script src="https://gist.github.com/JordiCorbilla/6a3f26571142eb7462e5092c37b58fca.js"></script><br />
The source code of the project can be found on my <b>github repository</b>:</div>
<div style="text-align: justify;">
</div>
<ul>
<li><a href="https://github.com/JordiCorbilla/BaaSDelphiSamples">BaaS Delphi Samples</a>.</li>
</ul>
<div style="text-align: justify;">
In this repository you will find simple examples using different vendors like <b>Parse.com</b>, <b>Kinvey </b>and <b>Firebase</b>, exploring different alternatives using HTTP Rest. You can read my previous articles about this topic here:</div>
<br />
<ul>
<li><a href="http://thundaxsoftware.blogspot.co.uk/2015/12/sending-rest-api-messages-with-delphi.html">Sending REST API messages with Delphi to Parse.com</a>.</li>
<li><a href="http://thundaxsoftware.blogspot.co.uk/2016/05/creating-your-own-self-hosted-parse.html">Creating your own self-hosted parse server</a>.</li>
<li><a href="http://thundaxsoftware.blogspot.co.uk/2016/05/deploying-parse-server-to-heroku.html">Deploying Parse server to Heroku</a>.</li>
<li><a href="http://thundaxsoftware.blogspot.co.uk/2016/06/baas-with-kinvey-and-delphi-101-berlin.html">BaaS with Kinvey and Delphi 10.1 Berlin</a>.</li>
</ul>
<br />
Firebase will store your documents in the Firebase Database in JSON format:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmpGZa94Uh1x5PF4_q5TE4j7UxyabH8x0-cjABMH5r4Zwb7C8pmWU19-H-cT51GvDJtUxUR4cd3HsZhGaaGj4SN48baQ9xStWrNSPjOND0ZtKUigilGMrzpiTCAGoqnxcmQ_YnWBJbML0/s1600/files.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="367" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmpGZa94Uh1x5PF4_q5TE4j7UxyabH8x0-cjABMH5r4Zwb7C8pmWU19-H-cT51GvDJtUxUR4cd3HsZhGaaGj4SN48baQ9xStWrNSPjOND0ZtKUigilGMrzpiTCAGoqnxcmQ_YnWBJbML0/s640/files.png" width="640" /></a></div>
<br />
<div style="text-align: justify;">
To use the Firebase Database URL as an end-point you just need to append .json to the end of the URL and send a request from our HTTPS component. For this example, I've created a project in Firebase called <b>delphitestproject</b>. Each request needs to be authenticated and the easiest way to achieve that is by attaching the <b>apiKey </b>from Firebase to your request. There are other options to authenticate your requests but these require exploring in deep firebase service accounts.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
To get the <b>apiKey</b>, go to <b>Settings </b>-> <b>Project Settings </b>-> <b>General</b>, and you will see there the web API Key:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6X6QzppAj0kngviMMjZJpRB8tUOnLQ5kgCIDpI8VVaWY5MxI1Ueskk6h4bWZVJMZYz-OTuV_8m5r3JfQkFcZf2Hv1V2hyphenhyphen4UwoZsVYRdeBv0oBRI8YXuGfIJF4KKw6gFJYSfPLcFbPGzY/s1600/apikey.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="233" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6X6QzppAj0kngviMMjZJpRB8tUOnLQ5kgCIDpI8VVaWY5MxI1Ueskk6h4bWZVJMZYz-OTuV_8m5r3JfQkFcZf2Hv1V2hyphenhyphen4UwoZsVYRdeBv0oBRI8YXuGfIJF4KKw6gFJYSfPLcFbPGzY/s640/apikey.png" width="640" /></a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
You can test your end-point using curl to simulate the requests:</div>
<div style="text-align: justify;">
<script src="https://gist.github.com/JordiCorbilla/8f66f10514f8504dafa9a69701ee26fc.js"></script></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The process to upload a file works as follows:</div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBKs4Gle8VIhPTEi0C2h1KnRZoEin2dvmahoRD2icYjIGhyphenhyphenH1_JYu2D593sBeiikzJPRqNXYaSnGVbAbZgnQHNHiJ6nbVaahYGtOt5LaciwznofHmPVSuvwGsMewfJGnFMu_wRxOEd_pE/s1600/Animation.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="406" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBKs4Gle8VIhPTEi0C2h1KnRZoEin2dvmahoRD2icYjIGhyphenhyphenH1_JYu2D593sBeiikzJPRqNXYaSnGVbAbZgnQHNHiJ6nbVaahYGtOt5LaciwznofHmPVSuvwGsMewfJGnFMu_wRxOEd_pE/s640/Animation.gif" width="640" /></a></div>
<br />
And the same solution running on my android and then opening the file from there as it has been downloaded from the cloud:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfmKB8yyViz81OudF9fiTMVzOfeiTsGPKkbISGfcQF6227UdgswYNULtaxslprK4aimecOOQOU7sKA7tAHNRFequSdn2KyG90JnAx2oZoEW6MXeLLzgtaggtjKJQltMSw-cquIkhGUXAY/s1600/Animation2.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfmKB8yyViz81OudF9fiTMVzOfeiTsGPKkbISGfcQF6227UdgswYNULtaxslprK4aimecOOQOU7sKA7tAHNRFequSdn2KyG90JnAx2oZoEW6MXeLLzgtaggtjKJQltMSw-cquIkhGUXAY/s640/Animation2.gif" width="356" /></a></div>
<br />
The Delphi app uses the the latest PPL (<a href="http://docwiki.embarcadero.com/RADStudio/Berlin/en/Using_the_Parallel_Programming_Library">Parallel programming Library</a>) to avoid freezing the UI. There are separate <a href="http://docwiki.embarcadero.com/RADStudio/Berlin/en/Using_TTask_from_the_Parallel_Programming_Library">TTask</a> that are called for the GET/POST request so there is no call in the UI Thread.<br />
When opening files on the Android app, I had to use Android intents to call the correct pdf component. The solution is still incomplete as I need to find a good way to browse files on my Android device and upload this one in the cloud database. From here onward, the sky is the limit. You could streamline the process by storing file Ids so then the whole content is not downloaded every time (which increases bandwidth usage) and encrypt your pdf files prior to streaming them to the cloud for an additional layer of security.<br />
<br />
Remember to explore the source code for more. Enjoy!<br />
<br />
<div style="text-align: right;">
Jordi</div>
</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com6tag:blogger.com,1999:blog-7022755517551007355.post-78861886098450410442016-10-04T00:51:00.001+02:002016-10-04T00:51:46.005+02:00Improve Code Quality with NDepend<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgILRt9Za7I1VJ7tkHzQt_kLXhbBVyYHf-wDvdSuqN1YA2AN5WTzG2dKi9OXqRtSIy4tt3RMv3ExfD_xN2doqXO9BBGHmwnv0mLmB2oefBIgZx-ImGVH2zuwvKvoxBnBH0_zlw_5hoS3_g/s1600/ndepend.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgILRt9Za7I1VJ7tkHzQt_kLXhbBVyYHf-wDvdSuqN1YA2AN5WTzG2dKi9OXqRtSIy4tt3RMv3ExfD_xN2doqXO9BBGHmwnv0mLmB2oefBIgZx-ImGVH2zuwvKvoxBnBH0_zlw_5hoS3_g/s200/ndepend.png" width="198" /></a></div>
<div style="text-align: justify;">
One of the most important things you could do to increase the code quality of your product and bring your team up to speed in terms of code reviews is to leverage the code quality analysis to a third party component. Nowadays I rely on <b><a href="https://msdn.microsoft.com/en-us/library/bb429476(v=vs.80).aspx">FXCop</a> (</b><i>a free static code analysis tool from Microsoft that checks .NET managed code assemblies for conformance to Microsoft's .NET Framework Design Guidelines. source: <a href="https://en.wikipedia.org/wiki/FxCop">wikipedia</a></i><b>)</b> to catch any code smell for me but this time I'm bringing an additional player into the game: <b>NDepend</b>.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>NDepend</b> "is <i>a static analysis tool for .NET managed code. This tool supports a large number of code metrics, allows for <u>visualization of dependencies</u> using directed graphs and dependency matrix. The tools also performs code base snapshots comparison, and validation of <u>architectural and quality rules</u>. User-defined rules can be written using LINQ queries. This possibility is named CQLinq. The tool also comes with a large number of predefined CQLinq code rules. Code rules can be checked automatically in <u>Visual Studio</u> or during <u>continuous integration</u>"</i>. source: <a href="https://en.wikipedia.org/wiki/NDepend">wikipedia</a>.<br />
<br />
<span style="font-size: large;"><b>Review your code!</b></span><br />
The best time to find any potential issues in code is as soon as the code is written. "<i>If you let code sit and rot for a while, it won't smell any prettier</i>"<i> by S. Venkat, H Andy on <a href="http://shop.oreilly.com/product/9780974514086.do">Practices of an Agile Developer</a>.</i><br />
<blockquote class="tr_bq">
<i>"Formal code inspections are about as twice as efficient as any known form of testing in finding deep and obscure programming bugs and are the only known methid to top 80% in defect-removal efficiency." by Capers Jones in <a href="https://books.google.co.uk/books/about/Estimating_Software_Costs.html?id=n5xQAAAAMAAJ&redir_esc=y&hl=en">Estimating Software Costs</a>.</i></blockquote>
As pointed out, code reviews are the best way to find and solve problems. Make sure you include a code analysis step during your continuous integration/continuous delivery model so issues are spotted earlier rather than later. Below you can see a simple pipeline that I tend to follow for any of my builds. This pipeline only refers to the commit stage and following the best practices to provide fast and provide useful feedback.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7VwHirPrLYhcFLoW6MTHLspqo4J9QU_LMCwtQLu5Nz_crIL6Iwo09f70C96HR2rBKea09gapyZiwvdNnSpwAjIMPaw_EPDwRqqkzcYLHVYDWWJKDQAxAfLtfqtJ1e-nPfZAjCrhE6cp8/s1600/graphn.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="380" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7VwHirPrLYhcFLoW6MTHLspqo4J9QU_LMCwtQLu5Nz_crIL6Iwo09f70C96HR2rBKea09gapyZiwvdNnSpwAjIMPaw_EPDwRqqkzcYLHVYDWWJKDQAxAfLtfqtJ1e-nPfZAjCrhE6cp8/s640/graphn.png" width="640" /></a></div>
<br />
When any code is checked-in to git, tfs, subversion, etc your build agent aka TeamCity, Cruise Control, Jenkins, etc. should pick up those changes and start processing that code through the pipeline. During this process, notifications should be sent to the user when something "breaks". In my case, anytime additional code duplicates or code smells increase during the build, the pipeline stops building and reports back to the user with the specific problem. I believe this is crucial to spot any possible issues in the code and that will help the developer to <u>take ownership</u> of the code.<br />
<br />
During the Code Analysis step, I rely on <b>FXCop </b>and <b>NDepend </b>to provide a <u>suite of principles and practices</u> that will make the code follow certain standards that are adopted by thousands of other developers. <b>FXCop </b>only provides a fair list of code inspections which are really useful but <b>NDepend </b>can give you more. <b>Let's see what NDepend can do for you!</b> (note that I'm assuming that you know FXCop quite well).<br />
<br />
<span style="font-size: large;"><b>Integrating NDepend with Visual Studio</b></span><br />
NDepend extension is really easy to install and integrate with your VS. Download the latest from <a href="http://www.ndepend.com/download">here</a> and follow the steps in this <a href="http://www.ndepend.com/docs/getting-started-with-ndepend">video</a> for its integration.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiauQNJN3cj1OHt_H3kHKrKKIiAXcffnF9QKYM0uKHq0s-h2xF-r7VfSW0i9BXPT5TOPyQmBYJ-WbCBGTpRfxT3qOckbMs1TfQY-BbhvpPpLfm-lWu8RLHJqeee3NZUb1nVER0TJzETHXk/s1600/ndepend2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="241" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiauQNJN3cj1OHt_H3kHKrKKIiAXcffnF9QKYM0uKHq0s-h2xF-r7VfSW0i9BXPT5TOPyQmBYJ-WbCBGTpRfxT3qOckbMs1TfQY-BbhvpPpLfm-lWu8RLHJqeee3NZUb1nVER0TJzETHXk/s320/ndepend2.png" width="320" /></a></div>
<br />
<span style="font-size: large;"><b>Integrating NDepend with TeamCity</b></span><br />
This is the bit that interests me the most. Thanks to the easy integration of <b>NDepend </b>with <b>TeamCity </b>I can leverage the code analysis step to focus on the final code review as the code should reach a <i>common quality standard</i>. To integrate NDepend with TeamCity, you can follow the steps in the following <a href="http://www.ndepend.com/docs/teamcity-integration-ndepend">tutorial</a>.<br />
<br />
Once configured you should see NDepend under your build runners:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEij09MYVNQ7TA-h7Yx8fDBmETaFe0ikPDZdsmEnBjEycJgG1tqT0L5r2fJgIUBCwMKG80gfTJc730u7Ha_gFlQIRbGqJUq-vReeL_xX1BGY5WfvLl_t3bjd9ttcGu_dtmia9vneQN15CEA/s1600/ndependtc.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="395" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEij09MYVNQ7TA-h7Yx8fDBmETaFe0ikPDZdsmEnBjEycJgG1tqT0L5r2fJgIUBCwMKG80gfTJc730u7Ha_gFlQIRbGqJUq-vReeL_xX1BGY5WfvLl_t3bjd9ttcGu_dtmia9vneQN15CEA/s640/ndependtc.png" width="640" /></a></div>
The next step is related to the project configuration itself. NDepend build step expects an NDepend Project file. To create one, you will have to do it in Visual Studio and check it in as part of the solution to your repository. Then TeamCity should see it as part of the source code.<br />
<br />
<b>Example of NDepend project as part of the source code:</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9o4d90I_XLk4Km988ToLW3mPTrUMrt8vMKHnuh-ur8guvU43A_z4bFUtCG4FKPigI13EjCnsHjroT2C6yEB0wlbSdRtWthUWvEYsOsOgXUUC7x-V-A6F3yl3TJqTMt069UVspO8oRdXc/s1600/projndepend.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9o4d90I_XLk4Km988ToLW3mPTrUMrt8vMKHnuh-ur8guvU43A_z4bFUtCG4FKPigI13EjCnsHjroT2C6yEB0wlbSdRtWthUWvEYsOsOgXUUC7x-V-A6F3yl3TJqTMt069UVspO8oRdXc/s640/projndepend.png" width="640" /></a></div>
<br />
<br />
To create a new project, go to VS and <b>NDepend -> Project -> New Project</b> and create a project under your main solution. Then you'll have to configure the dependencies that you want to attach to the project and once that's done you'll be able to explore the nice report that gets generated out of it.<br />
<br />
<b>Example of configuration:</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivlxWJk1sVdOmi44hF3CVJluQLwDl_uEwLfQv_DDwZnUEEr8fjHHJ7ZIx_xkM4bEPr-zUq2eB-WMzLgIEI2fmxMeLJqki4tT0HKGPfISEh1SzFSzXIH8l8dwDQ4ti4SFdAMpxSY8_DdTU/s1600/config.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="372" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivlxWJk1sVdOmi44hF3CVJluQLwDl_uEwLfQv_DDwZnUEEr8fjHHJ7ZIx_xkM4bEPr-zUq2eB-WMzLgIEI2fmxMeLJqki4tT0HKGPfISEh1SzFSzXIH8l8dwDQ4ti4SFdAMpxSY8_DdTU/s640/config.png" width="640" /></a></div>
<br />
<u>Notice that I had to open the project file to change the paths to <b>relative paths</b> as TeamCity works via relative paths and I was getting an error when running the project file</u>.<br />
<br />
Once you have your project configured, run Ndepend from your VS and generate your report:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaILtbC_LELLpLhrMj2W5ZC1P7nYkgWzzoKHxI3nrGR5CoU9povcbPB9_0sKhsoUAz9HcrQCVntMB9kmky9xKGzqJEB6kh8tM1IXQ9ZrFSyR_f3_fbpi4il3ChYbkRVjGWM107N2nhY7g/s1600/report.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="616" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaILtbC_LELLpLhrMj2W5ZC1P7nYkgWzzoKHxI3nrGR5CoU9povcbPB9_0sKhsoUAz9HcrQCVntMB9kmky9xKGzqJEB6kh8tM1IXQ9ZrFSyR_f3_fbpi4il3ChYbkRVjGWM107N2nhY7g/s640/report.png" width="640" /></a></div>
<br />
From this html Report, you can get all the summary metrics about your project and by clicking on them you can drill into the items to expand on those results.<br />
<br />
Once you configure <b>NDepend </b>in your <b>TeamCity </b>build step, you will get the same information with the amount of code inspections and errors that are encountered during the build:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix39sw9Yq_52h0RbpV61K67NJw4bDrxxS1j49ct7azNpdviPynDwBxZsKvh3LvIU40_id43lPI-ajL9ZWiBKWzcfdM4z4T9H1Grt1BUfFpISz5NVQKKVQchWe9KInH4Y-QkmIx8USC6jQ/s1600/tc1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="236" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix39sw9Yq_52h0RbpV61K67NJw4bDrxxS1j49ct7azNpdviPynDwBxZsKvh3LvIU40_id43lPI-ajL9ZWiBKWzcfdM4z4T9H1Grt1BUfFpISz5NVQKKVQchWe9KInH4Y-QkmIx8USC6jQ/s640/tc1.png" width="640" /></a></div>
The number of inspections appear on the build results so TeamCity can keep track of it overtime:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8TXc35TSyIvB1QfNxQSlScMLzvDQXy_u0YXetRb16XePrlKYCjJYdT5KqvOaUSoc-DOx_1_Iw0aE7KK2uan-cL25ZMwO-6ZxWGKgpeknRseLwr6lSakvaXHQKrCoBl6dqHLKfv0lFzfY/s1600/tc4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8TXc35TSyIvB1QfNxQSlScMLzvDQXy_u0YXetRb16XePrlKYCjJYdT5KqvOaUSoc-DOx_1_Iw0aE7KK2uan-cL25ZMwO-6ZxWGKgpeknRseLwr6lSakvaXHQKrCoBl6dqHLKfv0lFzfY/s640/tc4.png" width="640" /></a></div>
<br />
Once you drill into the build results, you will see the code inspections results (with all the findings from NDepend):<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgt2uIlzKc2bcwCYWxYMYcdfPXAMlDU9YvjBS7uceYsvc0IPdv6bQmYcimwLUPNYlZmFjHnLHRULfKZ8gBkbwbw0StqEDrDxYf-RVD2GTuTiQ9-cIMJSo8IBP8N_kZlaAv4j0h2gqt-wlI/s1600/tc2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="364" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgt2uIlzKc2bcwCYWxYMYcdfPXAMlDU9YvjBS7uceYsvc0IPdv6bQmYcimwLUPNYlZmFjHnLHRULfKZ8gBkbwbw0StqEDrDxYf-RVD2GTuTiQ9-cIMJSo8IBP8N_kZlaAv4j0h2gqt-wlI/s640/tc2.png" width="640" /></a></div>
And also the same report you saw in VS:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTe8ZY8JgFGZCZpIiBoJ8O8QaSm0nknD9wC3W_6zAcTZ6BppQUPex-oNs4W3H58N0u4jg9GvwvTn6DM5iIxAA1Plr1MFZ4r_5GAskNU2-Lq61baq7spP-afVF0El5BT8lq6tnGoNcoLxQ/s1600/tc3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="422" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTe8ZY8JgFGZCZpIiBoJ8O8QaSm0nknD9wC3W_6zAcTZ6BppQUPex-oNs4W3H58N0u4jg9GvwvTn6DM5iIxAA1Plr1MFZ4r_5GAskNU2-Lq61baq7spP-afVF0El5BT8lq6tnGoNcoLxQ/s640/tc3.png" width="640" /></a></div>
<br />
Now that our system is up and running and the integration of NDepend is done, we can focus on the important stuff...code quality.<br />
<br />
One of the things I like the most is the way code rules are set up. It's so easy to enable/disable rules that it's all done in the NDepend Project file. So if you are not agreeing with one particular rule for code quality, just disable it and it won't bother you anymore (try to disable one of the rules for FXCop...and you tell me how you did it and how long it took you..):<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhS_GfCmsqlQJvBjsXJDloG3Ny65golHZMOY9BuhNRMOo_Xj9rx8CtrdfEsiYORu9djd5uQI_N57VyJwW2cjoeMbJks4VU_6SfSNPFUB1GjnokwrrqVQUyg40zXe0kzY2QMpip-AIM9FXw/s1600/rules.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="348" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhS_GfCmsqlQJvBjsXJDloG3Ny65golHZMOY9BuhNRMOo_Xj9rx8CtrdfEsiYORu9djd5uQI_N57VyJwW2cjoeMbJks4VU_6SfSNPFUB1GjnokwrrqVQUyg40zXe0kzY2QMpip-AIM9FXw/s640/rules.png" width="640" /></a></div>
<br />
<a href="http://www.ndepend.com/docs/code-metrics">Here</a> the full list of code metric definitions.<br />
<br />
The main features of NDepend are listed below:<br />
<ul>
<li>Dependency Graph</li>
<li>Dependency Matrix</li>
<li>Treemap Metric View</li>
<li>Abstractness vs. Instability</li>
<li>Code Query</li>
</ul>
<span style="font-size: large;"><b>Dependency Graph</b></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj001B82UFAaT4-4TwNxeM_ROQPj8wdlWy416oS309Lju5-_wJByRo-7c-44afDFGVh2KdH7n6vN4PvdKH3U_t-UdWv_VFdQq0F5hDbhVxYrclosl95aAOM_tNab7zfnoRlGwXMGmDjtps/s1600/graph.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="348" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj001B82UFAaT4-4TwNxeM_ROQPj8wdlWy416oS309Lju5-_wJByRo-7c-44afDFGVh2KdH7n6vN4PvdKH3U_t-UdWv_VFdQq0F5hDbhVxYrclosl95aAOM_tNab7zfnoRlGwXMGmDjtps/s640/graph.png" width="640" /></a></div>
This very useful diagram allows you to see the relationship between different objects in your solution/s.<br />
<br />
<span style="font-size: large;"><b>Dependency Matrix</b></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEil80_VSQw7zD9ULP3jKU7VI26o21Rvb8q_8gqDp3v6Rc4QTSeONZCskBNU0B0nTTHdH7YyZhEUjPQvxHzo-6zBu3bvkz7JiM7MPdNwOZ2ViyBc2z1u9B1Ns_Biie9tFfGNFNkZrS3M3pU/s1600/matrix.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEil80_VSQw7zD9ULP3jKU7VI26o21Rvb8q_8gqDp3v6Rc4QTSeONZCskBNU0B0nTTHdH7YyZhEUjPQvxHzo-6zBu3bvkz7JiM7MPdNwOZ2ViyBc2z1u9B1Ns_Biie9tFfGNFNkZrS3M3pU/s1600/matrix.png" /></a></div>
The Dependency matrix gives you a coupling score matrix. This will help you identify highly coupled dependencies and if you click on the number, you can see the number of relationships for that particular entry:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmADhKqImVhyphenhyphenFeEJBX6W1RcL9rKxJ6hCq9gCtc6507Y9RQ0Zra_fAGJKkEgeColqWirrsDm0Ni8ASnszsP-OR4nX30RGnlsN9HiThsIf0rNkzYJN2zRoz-BE4B3zRV7oPRfMrKsgIVbls/s1600/depend.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmADhKqImVhyphenhyphenFeEJBX6W1RcL9rKxJ6hCq9gCtc6507Y9RQ0Zra_fAGJKkEgeColqWirrsDm0Ni8ASnszsP-OR4nX30RGnlsN9HiThsIf0rNkzYJN2zRoz-BE4B3zRV7oPRfMrKsgIVbls/s400/depend.png" width="400" /></a></div>
<br />
<span style="font-size: large;"><b>Treemap Metric View</b></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWaJaadVJquQFKds7WuWdgO0zkGFRm6CJQFa6hKtkg_PEaFoU4-TIFPQ2qoi7qsq4ByggjZKkPy9NOdKON43eZVc37HbjrADX53ua8wfxNRfwEmn2kzLAhyHFAgAWNAZ-jt3rY5_TpxmI/s1600/codemetric.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="372" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWaJaadVJquQFKds7WuWdgO0zkGFRm6CJQFa6hKtkg_PEaFoU4-TIFPQ2qoi7qsq4ByggjZKkPy9NOdKON43eZVc37HbjrADX53ua8wfxNRfwEmn2kzLAhyHFAgAWNAZ-jt3rY5_TpxmI/s640/codemetric.png" width="640" /></a></div>
In this spectacular view, your source code is mapped into nested rectangles showing methods that are relative to other methods. This will give you a great list of candidates for refactoring. In my case I have few boxes with a <a href="https://en.wikipedia.org/wiki/Cyclomatic_complexity">cyclomatic complexity</a> of 6 units which tell me that are good candidates for my review and analysis. Once you click on one of this rectangles the source code is shown.<br />
<br />
<span style="font-size: large;"><b>Abstractness vs. Instability</b></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSfJePsd83C1X-Zx5CqTITJUlhJw5qx9oTCIvviBJSTtoXf-ecFreoHSHD8_udWXMqedQ5ABzLp2rB7LVugyJZK8h8-7g1uLrSYLFfljwXgu-ArjY4_KjwpwqeysrSL4yfsAAuo0Xp0kM/s1600/abstract.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="379" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSfJePsd83C1X-Zx5CqTITJUlhJw5qx9oTCIvviBJSTtoXf-ecFreoHSHD8_udWXMqedQ5ABzLp2rB7LVugyJZK8h8-7g1uLrSYLFfljwXgu-ArjY4_KjwpwqeysrSL4yfsAAuo0Xp0kM/s400/abstract.png" width="400" /></a></div>
This graph give us a high level overview of our application in terms of where it is heading (too abstract or too unstable).<br />
<br />
Here is <a href="http://www.hanselman.com/blog/ExitingTheZoneOfPainStaticAnalysisWithNDepend.aspx">how to read this chart</a>:<br />
<br />
<ul>
<li><i>If an assembly is very stable (that is, lots of assemblies depend on it) and it's not extensible (no abstract classes, no virtuals, etc) then you're in the lower-left quadrant of the chart in the well-named <b>Zone of Pain</b>.</i></li>
<li><i>If an assembly is very abstract, very extensible, but no one depends on it (it's not really being used) then it moves towards the <b>Zone of Uselessness</b>.</i></li>
</ul>
<br />
<span style="font-size: large;"><b>Code Query</b></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPFz52HatnSaK84EHj2fXiiOMQ0rTeffxFXsHVNLXZU76m5SnDHsekjram8ryWPEsCzni0qNWy1FYu1jLAAXTs2uck4K-T2hRpqBqWpbCsXW4-WaUPEFdCk9rD_kj4gk4DU1OwLiAVdCY/s1600/code+query.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="608" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPFz52HatnSaK84EHj2fXiiOMQ0rTeffxFXsHVNLXZU76m5SnDHsekjram8ryWPEsCzni0qNWy1FYu1jLAAXTs2uck4K-T2hRpqBqWpbCsXW4-WaUPEFdCk9rD_kj4gk4DU1OwLiAVdCY/s640/code+query.png" width="640" /></a></div>
Code Query is NDepend tool that uses its scripting language (<a href="http://www.ndepend.com/docs/cqlinq-syntax">CQLinq</a>). This allows users to write their own queries and rules. I like the fact that it is highly customisable and developers love scripting components. I haven't played much with it but I can see the potential for it.<br />
<br />
<b style="font-size: x-large;">Conclusion</b><br />
I do believe that NDepend needs to be included in your tool-belt kit. Once you start using it you can't leave without it. Just the amount of information that handles for you is amazing and it gives you that level of confidence that all your developers are writing code with the same standards and code practices out there. It also helps you to spot complexity details that you might miss by just doing an informal review.<br />
<br />
Thanks to it's integration with TeamCity it made the decision easy for me as I rely a lot on catching everything during the commit stage. If any rule is broken, the build agent will inform the developer so he/she can take ownership of the committed changes and there won't be any argument about it. All the rules are set as part of the project and shared as part of the solution. <a href="http://www.ndepend.com/purchase">NDepend</a> is about $335 per developer license and about $671 for a build server license. It is highly documented and I found all the help I needed online..the guys from NDepend did really a great job!.<br />
<br />
<b>Full Disclosure</b>: I received a complimentary copy of NDepend to provide a review from my point of view. I'm a working professional with loads of years of commercial development experience with hundreds of hours (not to say millions) of code reviews on my shoulders and I don't have time to use useless tools and certainly this is not the case. NDepend is extremely helpful and a great addition to be considered by any development team out there.</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com1tag:blogger.com,1999:blog-7022755517551007355.post-68048497027541963352016-09-06T23:29:00.000+02:002016-09-06T23:29:13.687+02:00Delphi REST client that talks to a REST Web API on a Raspberry PI in JavaScript<div style="text-align: justify;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidPlKTOJl3cxsQJjvlhxg8zH6KGpy-wj6BsqAEwkbj9nGewCantgnJJU1nSGQDJq6HNBddBKe-i4DwdgP04eDY64mbAhBS-Li1XG7K-1oWtHRucnNmKPmimGgSyqw7pQIIzcLd7x1ghC0/s1600/status.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="179" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidPlKTOJl3cxsQJjvlhxg8zH6KGpy-wj6BsqAEwkbj9nGewCantgnJJU1nSGQDJq6HNBddBKe-i4DwdgP04eDY64mbAhBS-Li1XG7K-1oWtHRucnNmKPmimGgSyqw7pQIIzcLd7x1ghC0/s200/status.png" width="200" /></a><span id="goog_1560995759"></span><span id="goog_1560995760"></span>This article presents an approach for developing a mobile application to monitor a Raspberry Pi cluster using a simple Web API built in JavaScript. If you haven't followed my progress regarding the Raspberry Pi cluster you can read my article <a href="http://thundaxsoftware.blogspot.co.uk/2016/07/creating-raspberry-pi-3-cluster.html">here</a>. I would also encourage you to read the article regarding node.js and how I created a simple <a href="http://thundaxsoftware.blogspot.co.uk/2016/08/simple-website-in-nodejs-for-you.html">web monitor</a>.<br />
<br />
The main idea is to be able to check the status of my grid from anywhere and the best place for it is on my mobile device. In this case I extrapolated the idea from my initial article so I could reuse some of the concepts and leverage the User Interface with <b>Delphi 10.1 Berlin</b>. As I'm still waiting for the <a href="http://community.embarcadero.com/article/news/16211-embarcadero-rad-studio-2016-product-approach-and-roadmap-2">Delphi version that offers Linux Server</a> support I will have to use a <a href="https://en.wikipedia.org/wiki/Representational_state_transfer">RESTful</a> approach to communicate with my devices with ease.<br />
<br />
In this case, the Android application will talk to the <b>REST Web API</b> that sits in one of the Raspberry PIs. This Web API is built using <b>node.js</b>. Here is the architecture of the example to build:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbtdznFwUK060UPhBoJ0OOi3_DGuiwEXKFQZDRqVIAEtCo4uWa9HddHxnScz4EEbVOLXAyxOt_8ltE5VmM6K9jOP3VBi1KLgvEgFN7cRkI8VpcEn3jYMx4LFKCh1ai4xjgM3xx_iOVmhw/s1600/architecture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbtdznFwUK060UPhBoJ0OOi3_DGuiwEXKFQZDRqVIAEtCo4uWa9HddHxnScz4EEbVOLXAyxOt_8ltE5VmM6K9jOP3VBi1KLgvEgFN7cRkI8VpcEn3jYMx4LFKCh1ai4xjgM3xx_iOVmhw/s1600/architecture.png" /></a></div>
<br />
I have a cluster of 4 Raspberry Pi and I want to check if they are <b>alive</b> without having to go physically to them (I saw few times where those little ones were not responding properly after a reboot). Thanks to the RAD that Delphi 10.1 Berlin offers I can have my Android application up and running in less than one hour as the components are already in-built and I only have to worry about the design and the responsiveness of the final application.<br />
<br />
On the back-end, I have a really simple JavaScript Web API built with node.js. This simple API awaits external requests through a controller named status which expects machine names (e.g. <b>pi02:3000/status/pi01</b>) or IP addresses (e.g. <b>pi02:3000/status/192.168.1.1</b>) to test the connectivity to them. The result is in json format. The source code can be seen below:<br />
<script src="https://gist.github.com/JordiCorbilla/5fce6fc27682e33f6b39efeb955b60f4.js"></script><br />
Alternatively, the source code of the entire project can be found in Github at the link below:<br />
<ul>
<li><a href="https://github.com/JordiCorbilla/Delphi-RaspberryPi">https://github.com/JordiCorbilla/Delphi-RaspberryPi</a></li>
</ul>
In here you will find the Web API and the Delphi Application (<b>Win64 </b>and <b>Android</b>). You could also run it in iOS but this is up to you to test.<br />
<br />
Here the Web API in action:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHNQ0xkRHkfg3vhrCFLDrZTYzzFuUO2RSccJ7iihW0Kbt1g2Js6KVXnH-fk_aP6QB1c_TMKPzPwz9rHBgupT4_d74vWbaIaJg5F9Vs94W_i3l_kvik62kqlZFkieLSrdfejra4QMxuIKY/s1600/Animation.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHNQ0xkRHkfg3vhrCFLDrZTYzzFuUO2RSccJ7iihW0Kbt1g2Js6KVXnH-fk_aP6QB1c_TMKPzPwz9rHBgupT4_d74vWbaIaJg5F9Vs94W_i3l_kvik62kqlZFkieLSrdfejra4QMxuIKY/s640/Animation.gif" width="640" /></a></div>
As you can see it's very simple. The API just pings the device you ask for and returns <b>dead/alive</b> in <b>json</b> format. This is then parsed by the client and represented in different statuses. To ping the devices I'm using the <a href="https://www.npmjs.com/package/ping">ping npm</a> package.<br />
<br />
The layout of my Android application using Delphi is as follows:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO8VRFUBEuF8j3hYjLo6vkz1yceVEQk7wQlNqEcmRLTgoTxZ5Z7iCKNGZgxLAOiCAY0JlunxDAu7HGSArCEay0EJgWEhMjgy87ztNa_5JH85e9FHpQz-VmWFxFgNzS7s8UxxcyzIS0mxg/s1600/layout.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="387" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO8VRFUBEuF8j3hYjLo6vkz1yceVEQk7wQlNqEcmRLTgoTxZ5Z7iCKNGZgxLAOiCAY0JlunxDAu7HGSArCEay0EJgWEhMjgy87ztNa_5JH85e9FHpQz-VmWFxFgNzS7s8UxxcyzIS0mxg/s640/layout.png" width="640" /></a></div>
<br />
As you can see I'm using the components <b>TRestClient</b>, <b>TRestRequest </b>and <b>TRestResponse</b>. Now I need to be able to send REST requests to my WebAPI. To do this I need to configure these three components in the following way:<br />
<br />
Place the three components in your form and they will automatically reference each other, then configure the <b><a href="http://docwiki.embarcadero.com/RADStudio/Seattle/en/REST_Client_Library">TRestClient</a></b> using the following parameters:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQjD6Oh9z1fghc8MdOasqCOlsO6yHR7dEXu26tfAK8wu31ymKoZ_ko4MQOrl5r8Ml7rNV8c2d6qwjTitPHpr8L6RFcSXnsr08JEzv45pEg2MgVteGtoW7pz4j0jz0bJtz44eXz61HP0PQ/s1600/restclient.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="396" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQjD6Oh9z1fghc8MdOasqCOlsO6yHR7dEXu26tfAK8wu31ymKoZ_ko4MQOrl5r8Ml7rNV8c2d6qwjTitPHpr8L6RFcSXnsr08JEzv45pEg2MgVteGtoW7pz4j0jz0bJtz44eXz61HP0PQ/s640/restclient.png" width="640" /></a></div>
<br />
The important ones are <b>Accept</b>, <b>AcceptCharset</b> and <b>BaseUrl</b> (<i>this last one will contain the url you want to request</i>). Notice that this is just the base URL as the resource will be specified in the <b>RestRequest</b> component.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAhyVrXnpRzeO0YGHFrmGY_VQiu2NO3QS4lNSZdyNlBO_XGVPpujsZ57ebUwBV0l-T8feWWOHZ5Of_yr2SPmzWSjBuSpgSYrnGdxNX8GyHMKBMeAfI_LjuuzjvjFQUvo0-HmGe3oErI_U/s1600/restrequest.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="356" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAhyVrXnpRzeO0YGHFrmGY_VQiu2NO3QS4lNSZdyNlBO_XGVPpujsZ57ebUwBV0l-T8feWWOHZ5Of_yr2SPmzWSjBuSpgSYrnGdxNX8GyHMKBMeAfI_LjuuzjvjFQUvo0-HmGe3oErI_U/s640/restrequest.png" width="640" /></a></div>
Now you only need to edit this "Resource" property and use the correct argument to check the machine you want to check.<br />
<br />
To make sure that the application is fully responsive I'm using a different thread to check the communications and also show a nice <b>TAniIndicator</b> while the operation occurs. This will make the application fully responsive without freezing the app.<br />
<br />
Here is a summary of the code for your review:<br />
<script src="https://gist.github.com/JordiCorbilla/69fb48d7e48b70564191f4d6d6b64313.js"></script><br />
You can find the entire source code here:<br />
<ul>
<li><a href="https://github.com/JordiCorbilla/Delphi-RaspberryPi">https://github.com/JordiCorbilla/Delphi-RaspberryPi</a></li>
</ul>
Finally the application in action:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEie6lcDogBG9P3ANwaFitNcjsAG-6JB4ZWW2yrgo0O0_6R89rH2FtLqbKqg1_FDbohbGGyzB4UCi8x3K8tk61NVMItC1Z-_qPXHNYhVxS9YPVBD5E0YEWiDctxjzAUPjX-eap9Kl34WP2k/s1600/Animation2.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="618" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEie6lcDogBG9P3ANwaFitNcjsAG-6JB4ZWW2yrgo0O0_6R89rH2FtLqbKqg1_FDbohbGGyzB4UCi8x3K8tk61NVMItC1Z-_qPXHNYhVxS9YPVBD5E0YEWiDctxjzAUPjX-eap9Kl34WP2k/s640/Animation2.gif" width="640" /></a></div>
<br />
If you wonder :), I'm using <a href="https://chrome.google.com/webstore/detail/vysor/gidgenkbbabolejbgbpnhbimgjbffefm">Vysor Chrome app</a> to mirror my android device and <a href="https://screentogif.codeplex.com/">ScreenToGif</a> to generate my Gifs.<br />
<br />
You will find all the necessary details on my Github project (installation procedures on the Raspberry Pi, etc).<br />
<br />
I look forward to your comments.<br />
Jordi</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com1tag:blogger.com,1999:blog-7022755517551007355.post-77075308202494774812016-08-21T11:58:00.001+02:002016-08-27T16:39:55.181+02:00Simple Website in Node.js for you Raspberry Pi 3<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixvWmlMNukSya9RAFjIBg2AnGdDkYSSRcSoI5XX6P-C34vCxEa7qzYjU7J0lDpb_uDWcjxq8YR0XjAQ5qCvz_WiWpogTWGW4G_8FmK6jBdEbE3ZE0TCjRefZvUcJcQnpWE3_UWDpiKsMQ/s1600/website.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixvWmlMNukSya9RAFjIBg2AnGdDkYSSRcSoI5XX6P-C34vCxEa7qzYjU7J0lDpb_uDWcjxq8YR0XjAQ5qCvz_WiWpogTWGW4G_8FmK6jBdEbE3ZE0TCjRefZvUcJcQnpWE3_UWDpiKsMQ/s1600/website.png" /></a></div>
<div style="text-align: justify;">
Now that I have a <a href="http://thundaxsoftware.blogspot.co.uk/2016/07/creating-raspberry-pi-3-cluster.html">cluster of Raspberry Pi</a>'s my possibilities are <u>endless</u>. In this article I will show you how to create a simple website using <b><a href="https://nodejs.org/en/">Node.js</a></b> with <b><a href="https://expressjs.com/">Express</a></b>, <b><a href="http://stylus-lang.com/">Stylus</a></b> and <a href="https://www.npmjs.com/package/pug">Pug</a>. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Node.js</b> is is a platform built on Chrome's JavaScript runtime for easily building fast and scalable network applications, <b>Express</b> is a fast web framework for Node.js, <b>Stylus</b> is an innovative style-sheet language that compiles down to CSS and <b>Pug</b> is a succinct language for writing HTML templates.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
To ease the pain of working with a Raspberry Pi, I will show you how to remote desktop it first for ease of use so you can code your website on the Pi itself without having to use the console.<br />
<br />
The following operations have to be applied to every member of the cluster so Remote desktop is available to every node. The idea is to access each node via remote desktop using a laptop with windows 10.<br />
<h2>
Configure Remote desktop on the Raspberry Pi</h2>
The first thing to do is to update our version of <b>Raspbian </b>and ensure that all the packages are upgraded. This can be done with the following commands:<br />
<br />
<b><span style="color: blue;">sudo apt-get update</span></b><br />
<br />
Next, run the following command to upgrade any packages installed on your system that need upgrading:<br />
<br />
<b><span style="color: blue;">sudo apt-get dist-upgrade</span></b><br />
<br />
<u>You'll have to perform this operation in all your cluster nodes</u>. Now we are ready to install the packages we need: <b><a href="http://www.xrdp.org/">xrdp</a></b> and <b><a href="https://www.samba.org/">samba</a></b>. xrdp is an open source remote desktop protocol(rdp) server and Samba is the standard Windows interoperability suite of programs for Linux and Unix.<br />
<br />
Run the following commands so we can remote desktop the Raspberry Pi's:<br />
<br />
<b><span style="color: blue;">sudo apt-get -y install xrdp</span></b><br />
<br />
The '<b>-y</b>' option will automatically answer <b>yes</b> to the default <b>continue [Y/n]</b> question.<br />
<br />
Next step is to install the <b>samba</b> package so we will be able to access the Raspberry Pi's by its host name from Windows rather than by it’s IP address which changes as the node receives its IP address via DHCP:<br />
<br />
<b><span style="color: blue;">sudo apt-get -y install samba</span></b><br />
<br />
After the installation is successful, you should be able to ping the Raspberry Pi from your windows machine and perform the remote desktop:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKUJXMwRAvN3V4QJNLEngFSlnvWRvSSAUF_2KFb33owjKdKSqgB6fmxmVa5zzy6Jx6tdXhD1y1k4Wr1x5zKbcEEoiJpoxhS6KPsQzwv0QXJGm0VniBNMGfQmamEoos6c4edse7GE_GufA/s1600/remote.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKUJXMwRAvN3V4QJNLEngFSlnvWRvSSAUF_2KFb33owjKdKSqgB6fmxmVa5zzy6Jx6tdXhD1y1k4Wr1x5zKbcEEoiJpoxhS6KPsQzwv0QXJGm0VniBNMGfQmamEoos6c4edse7GE_GufA/s1600/remote.png" /></a></div>
<br />
If everything is configured correctly you should see the following screens:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPa5QXzpY_vkWnMSU5IP-g2hHAh67-MYvi5UpBSYBtjfsR2PVBUph92GS4APshSXjHH0_vTIMvSw_6LqYvedeOYYOVOtj1h-TVGZDOK3oa3ywiYgFx3jilJSSo7GPML5wFqcOD-9LuTdY/s1600/xrdp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPa5QXzpY_vkWnMSU5IP-g2hHAh67-MYvi5UpBSYBtjfsR2PVBUph92GS4APshSXjHH0_vTIMvSw_6LqYvedeOYYOVOtj1h-TVGZDOK3oa3ywiYgFx3jilJSSo7GPML5wFqcOD-9LuTdY/s1600/xrdp.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgphXMPCkE5oPS4QdxO8Y99-Y7pWQ_0yr9PNlkiuURsCiOkYfyhHAIoDm4lVVIc95mfNwg_qCKK635GgkJf8oIYnV7REZPZDURPcf7Zw0P7dRKKMZ-w1aCxjauEN6urj7gmGUmb7gh_-18/s1600/log.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgphXMPCkE5oPS4QdxO8Y99-Y7pWQ_0yr9PNlkiuURsCiOkYfyhHAIoDm4lVVIc95mfNwg_qCKK635GgkJf8oIYnV7REZPZDURPcf7Zw0P7dRKKMZ-w1aCxjauEN6urj7gmGUmb7gh_-18/s1600/log.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBomhAQ-F9Qq51fcIhQny3dFJF8slGnDMKasyq46q9cMr_dfoLMA4Pmgj36CSCoLEbhYYQ6i1p6bISxLJkYx6OkW-JEx26zqvcOMXy_XWZe1616hty5K5YVZRCzwePyzp4lI18Zgk0D0U/s1600/remotepi.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBomhAQ-F9Qq51fcIhQny3dFJF8slGnDMKasyq46q9cMr_dfoLMA4Pmgj36CSCoLEbhYYQ6i1p6bISxLJkYx6OkW-JEx26zqvcOMXy_XWZe1616hty5K5YVZRCzwePyzp4lI18Zgk0D0U/s640/remotepi.png" width="640" /></a></div>
<br />
To copy files from my Win10 machine to one of the nodes I use <a href="https://winscp.net/eng/download.php">WinSCP</a>. You can also create a shared folder on the Pi that's visible on your Win10 machine using <b>Samba</b>.<br />
<br />
Now it's time to configure the rest.<br />
<h2>
Install Node.js</h2>
By default there is a pre-installed version of Node.js on the Raspberry Pi's. If you type <b>node -v</b> you'll see that the version of node.js is <b>v0.10.29</b>. We need to upgrade this version to a more recent one (<b>v6.3.1</b> by the time I published this article).<br />
<br />
<script src="https://gist.github.com/JordiCorbilla/01810a7b0f0953f43bfe52cd640fe112.js"></script>
Now our Raspberry Pi is ready for action. Let's see what are the next steps to create our Website.<br />
<br />
To make things easier for you, I've created a project on <b>Github</b> that contains a <a href="https://github.com/JordiCorbilla/ExpressSiteExample">sample website</a> that you can use to start with. Installing the dependencies required through <b>npm</b> is a bit cumbersome so using a sample project makes things a bit easier.<br />
<br />
Here is a screenshot of the site once it is up and running:<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg95nDtfY8l0gOknZxbatM8Zq6xdm_0yQNpxqjik2yDpB_Bfys6n_Zbrdg7M35uiRhjvzR_BoBb7_-gITp_R9zhHc659saywuPXu06AiY8E-prHJEEdgVVSeirDUheB_XsbO1A9OtPde6U/s1600/rdppi.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg95nDtfY8l0gOknZxbatM8Zq6xdm_0yQNpxqjik2yDpB_Bfys6n_Zbrdg7M35uiRhjvzR_BoBb7_-gITp_R9zhHc659saywuPXu06AiY8E-prHJEEdgVVSeirDUheB_XsbO1A9OtPde6U/s640/rdppi.png" width="640" /></a></div>
<br />
I'm using the site as a Raspberry Pi status monitor where there is a bit of javascript that pings each node on the grid. Then I use <a href="http://knockoutjs.com/">knockoutjs</a> to bind the results to the page.<br />
<br />
Once you've downloaded the repository, you only need to run the following commands to install the dependencies and run the website:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1fTBD1q7Ve_d7pKEZhV_khMGUpDrBOpXMQE1Gz8fIZJglE9enEGN91okkyRRnU8lOQFElOI5US9W4ELyHE5zQk4hyA1Ovgjc-lm19R6XqFacSabVFUzZ4VUAblLySVjcIy7btt9mNXng/s1600/runningweb.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="420" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1fTBD1q7Ve_d7pKEZhV_khMGUpDrBOpXMQE1Gz8fIZJglE9enEGN91okkyRRnU8lOQFElOI5US9W4ELyHE5zQk4hyA1Ovgjc-lm19R6XqFacSabVFUzZ4VUAblLySVjcIy7btt9mNXng/s640/runningweb.png" width="640" /></a></div>
You can follow the instructions on my Github project:<br />
<ul>
<li><a href="https://github.com/JordiCorbilla/ExpressSiteExample"><b>ExpressSiteExample</b></a>.</li>
</ul>
The site in action:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5hwoYA96YUL8q1wTDFJwq_cxxhLxmAQP9Puh-_vZEeC3u7vAh78LDQCbLQ1LypvGrpBSgueLN5ZsOPk35cB6CkNmhTxJxK6ssVwcx0EEFwJCu_u580755r_wsTTE5VPqvhW8307Y_wis/s1600/rpi.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="490" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5hwoYA96YUL8q1wTDFJwq_cxxhLxmAQP9Puh-_vZEeC3u7vAh78LDQCbLQ1LypvGrpBSgueLN5ZsOPk35cB6CkNmhTxJxK6ssVwcx0EEFwJCu_u580755r_wsTTE5VPqvhW8307Y_wis/s640/rpi.gif" width="640" /></a></div>
<br />
Here is the list of installed packages for your reference:<br />
<script src="https://gist.github.com/JordiCorbilla/e4436be0f0ee3abca62a5958ca8918e5.js"></script>
<br />
<br />
<div style="text-align: right;">
Jordi.</div>
</div>
Jordi Corbillahttp://www.blogger.com/profile/15899608947597831029noreply@blogger.com1