Information Technology Strategies in Software Development Projects

The goal of software projects is to complete them on time and on budget. There can somewhat be an added challenge, when part of the project is also to create specifications for hardware and software acquisitions that are needed to deploy a solution.  In the myriad of possible configurations and architectures that advances in Information Technology have created, there can sometime be no clear path as to what should be budgeted, and if the budget is even adequate.

Licensing can a large obstacle to having a successful software project.  How third-party software can be used and deployed (for example: the number of connections or users allowed by license cost) is just one example of the difficulty in obtaining proper budget projections.  Add to this the complexity of the type of hardware needed and the specifications of that hardware.  Not only do companies have to spec hardware for their own internal development practices (if they do not have the resources beforehand), clients will also put into a statement-of-work (SOW), that the vendor create the correct specifications to deploy the software on site or in the cloud or both.

Information technology needs to become a project planning process rather than an line item on a project Gantt chart.  Creating a road map where IT can become a process that has a business component at every step, will create significant predictability in software projects with less risk.  Strategies such as IT Governance, Unified Modeling, hardware sizing, use-case analysis and economic cost analysis are essentially putting pen and paper together to plan all aspects of IT prior to planning any budget or acquiring any product.

If you’d like more information or to ask questions on how create IT Business Strategies, please feel free to contact me.

 

Designing and Building a Self-Driving Car – Part 4

Topics:  The many lessons of Pulse Width Modification or “Where have I seen this before?” Thoughts on Motor Speed and where to look for inspiration for a project.

This blog will be about the many lessons one learns throughout his or her life – and boy! have I’ve learned many!  In college I took electrical and electronics engineering courses, but one of the biggest lessons learned were not about resistors, capacitors, or Kirchhoff’s Voltage Law, or transistors; it was: ” You will forget what you learn quickly, if you don’t apply it in practice.”  This is the lesson that I intend to instill in my child throughout this project.

My “Parent’s Guide” tip is to create a project that will continuously reinforce the concepts that you will be using for the project.  The best way, in my opinion, to do this is to have a project that will last many months (or in years in my case).  Take a detour now and then to do a science fair or demonstration in from of the grandparents, science clubs and assortment of friends; in between boy scouts, or girl scouts, baseball, volleyball, tennis, swimming, or the myriad of activities children do throughout their lives, you need one project that is consistent and continuous and big for which they can take into college or whatever the next big step in their lives will be.

Learned about pulse modifications throughout my education.  First in an electronics lab course in 1993, and again in 1997 and 1999.  When I began working on the autonomous car project with my child, we began working on the PWM circuit for the electric motor that was to go into the car.  I forgot how it worked or what it would take to build this circuit.

The morale is working on a project such as this she help to continuously reinforce concepts of STEM throughout a child’s life and should not just be about getting it done!

We decided to create multiple prototypes (4), varying in difficulty, size, configuration and purpose.

Prototype 1:  Will be the an simplified Arduino and RaspberryPi prototype built for the sole purpose of controlling a DC Motor and a Servo with OpenCV and a RaspberryPi camera.  It’s a non-functional RC model.

IMG_2630[1]

Prototype 2:  Will be a workable RC Car that will do everything that Prototype 1 will do, but will have a Brushless motor with ESC and will move.

IMG_0849

Prototype 3:  Will be a 3D printed model designed in Bender and then 3D printed.  This prototype will better emulate an “actual car”.  It will have all the essentials such as a transmission, steering, suspension, differential, axle, etc.  It will not have all the other things a car has that have nothing to do with power, acceleration or steering.  I will be a working model with an actual brushless motor, computers and microcontrollers.  It will not have cameras, but will be operated by a dataset from actual road video.

2019-10-29_16-56-33

Figure 1:  Blender open source design software

Prototype 4:   An actual gasoline car rebuilt into an electrified self-driving car.  My hope is to rebuild an old Geo Metro.  But the chances are a little small at this point, because Geo Metros all over the country are being snapped up to convert to electric.  The advantages of rebuilding an electric car using a Geo Metro are:

  1. They’re cheap.  Many have bad engines which make them even cheaper.  You could find one from $300 to $1500.
  2. They’re lightweight.  Geo Metros are three cylinder and don’t have all the stuff that weights modern cars down.
  3. The shape is conducive to the space you would need for an electric car.  Since the car that I’m building will also contain computers and other mechanisms to make it self-driving, the hatchback is particularly attractive.

 

Chevrolet_Metro_hatchback_rear

 

Information Technology Management Strategies in Industry Series

Use cases for Information Technology in the energy and manufacturing industry continue to expand beyond typical financials, asset management and plant management applications. Now IT is being use more often in driving goals such as resourcefulness and efficiency in current business processes and products.

Starting next month until May, I will be releasing a series of blogs addressing IT Strategies in areas of business and Industry in manufacturing and energy.  The series will include:

  • Information Technology Management Strategies for Energy Management
  • Information Technology Management Strategies for Customer Service 
  • Information Technology Management Strategies for the Internet of Things and Smart Cities Initiatives 
  • Information Technology Management Strategies for Data Analysis in Manufacturing

The main objective of this series is to apply IT management knowledge to experiences in energy, manufacturing and customer service.  With the advent of new innovative solutions in areas such as data science and machine learning, there are more opportunities than ever to make these industries more resourceful, efficient and effective in its business processes and beyond.

I will post links to the blogs from this page for easy reference in the future.

Effectively using Associative Tables in Relational Database Design

 

When creating entities in an entity-relational (ER) diagram, there are times where multiple entity attributes (also known as fields in tables) have associations to could create redundancy in the entity instances of a relational database model.  An associative entity is an entity type that associates the instances of one or more entity types that contains attributes in the relationships between those entities.   Associative entities  helps prevent joining two or more entities directly with multiple relationships that would create duplication of instances.  In databases this can sometime result in Cartesian joins and can produce duplication in table rows; sometimes many times over in a result set.

In entities where the instances are not very unique, such as the days of the week, months in a year, and positions in a company; it’s not a good idea to create a direct association with attributes on that entity to another entity that has very unique instances.  For example, meetings in a company, full names, age and address of people in New York City, or people working in a company are examples of uniqueness.  Although these are a very basic example, you will soon discover that you could almost create a level of duplication that can exponentially grow the number of instances in the design of your database model.

Below is an use case for an associative entity.  The EMPLOYEE  and SKILLS tables are the entities that contain the employee information and types of skills and titles within the company, respectively.  In this example a company ranks it’s employee’s skills by titles such as Principal, Senior, III, II,  and I.  However, they also want to know which skills are managed by which employees.  Since we want to prevent as much duplication as possible, we create an associative table called ORGANIZATION that will have a primary identifier attribute (ORG_ID) and joins together the EMPLOYEE and SKILLS table. The MANAGED_SKILL_ID attribute would create a composition identifier that would allow an employee to have multiple instances of the skills for which he or she would manage without duplicating data in the EMPLOYEE or SKILLS entities.

2019-11-10_6-13-37

 

Top Data Analytics and Data Science Resources

Here are my favorite Data Science/Data Analytics Resources

 Curriculum

1) MIT Open Courseware 

A great MOOC (Massive Online Open Courses) to learn about the math and statistical fundamentals of Data Science, such as linear algebra, statistics, probability, etc. From a
great university. This will give you some of the fundamentals of data science. https://ocw.mit.edu

2) https://kaggle.com

A website that sets up data analytics and data science competitions. It also provides a lot of free data that you can play build your skills on.

3) https://Datasciencemasters.org

A open-source curriculum for learning Data Science. Including foundation in theory and technologies. You can download code and use it to build
project to improve your skills.

4) https://superdatascience.com/pages/machine-learning

I really like like this website, because it teaches you all the fundamentals of machine learning from A to Z.

5) https://Coursera.org

A MOOC (Massive Online Open Courses) that can teach you anything you need to know about Data Science and Machine Learning and Data analytics

6) https://Udemy.com

Another good MOOC

7) https://EdX.org

Another good MOOC

8) Standford Online (https://online.stanford.edu/)

A lot like MIT Open Courseware. Free and from a renowned university.

Programming

4) https://Anaconda.com

This is my favorite Data Science/Data Analytics platform. Python is very hot right now in regards to Data Science. Anaconda is the best platform to learn and program in Python.
Strongly recommend learning Python and/or R. It looks like you’ve learned SQL, which is the other popular language.

5) Sci-kit Learn (https://scikit-learn.org)

This is my favorite library for data science and deep learning. A lot of great features for classification and anomaly detect and other stuff.

6) https://Github.com

Github is a community source repository for Python, C++, C#, Java, and Javascript. You should create your own GitHub account and start being active on it. There are a lot of tutorials.

7) https://www.w3schools.com/

You can learn almost any coding language here. C++/Python

There are a lot of good books from Amazon to learn Python.

Datasets

My Favorite Publicly Available Datasets (also see https://rtpopendata.com/2019/02/03/my-favorite-publicly-available-datasets/)

I’ve been working with data for decades, searching for insights, converting it, managing it, and now performing data analytics. We have access to unbelievable treasure troves of public data to analyze. Many of the blogs I write are based on these datasets, as I don’t have access to large computing systems. Here is a list of my favorite publicly available datasets. Enjoy!

PJM Interconnection Data Dictionary for electrical grids, distribution and transmission. https://www.pjm.com/markets-and-operations/data-dictionary.aspx
University of California Irvin (UCI) has a huge machine learning repository to practice techniques. This repository can be accessed at archive.ics.uci.edu/ml/index.php
Amazon Web Services datasets are available to the public. https://aws.amazon.com/datasets/.
Kaggle is a data science competition website that rewards prizes to teams for the best ML models. Datasets are located at https://www.kaggle.com/datasets
University of Michigan Sentiment Data.
The time series data repositories are located at https://fred.stlouisfed.org/categories.
Canadian Institute of Cyber Security. https://www.unb.ca/cic/datasets/nsl.html.
Datasets for “The Elements of Statistical Learning”. https://web.stanford.edu/~hastie/ElemStatLearn/.
Government Open Data Portal. https://data.gov

Parent’s Guide to Designing and Building a Self-Driving Car with Their Kids – Part 3

“It’s not the destination, it’s the Journey…” 

I suspect building a full-size self-driving car seems like a momentous task – and it is.  But a few things we have going for us.  As I stated in my previous blog, autonomous (self-driving) cars have a lot of the STEM aspects you want to instill in your child – math, science, electronics, and technology.  So even if you don’t finish this, those subjects will take your child far.

A few things I would recommend is building a design for you car.  The cheapest way of designing anything is using a Computer-Aided Design (CAD) software.  For me, this is Autodesk® AutoCAD® 2017.  It’s a great software package, but a little on the pricey side.  There are also plenty of open source CAD software packages available.  The nice thing about AutoCAD is that i comes with an add-on called Autodesk EAGLE, which is a electronics schematic design tool.  Inevitably, there will be some electronic circuits required to build the prototype and eventually the actual car, so having an electronics design tool will be very helpful.

2019-09-02_7-25-46

Autodesk EAGLE, a electronics design tool

I alluded this earlier in one of my blogs, but you will want to build a prototype that makes it easier for kids to learn about the curriculum and about the subject matter involved. A prototype has a smaller budget an can be a much smaller than the eventual final product.  In my case I took apart one of my child’s toys and hooked it up to a RaspberryPi and Arduino (see Part 2 for more info).

Having a cash budget and setting design limitations on the car will take out a some of the risk of a venture such as this.  For instance, our stated goal was to create a car that would not have an occupant riding inside of it, nor would it be on any public roads. This would not require us to get specific permits or spend heavily on safety features of the car. Before building anything large, my recommendation is to have the following:

  1. A budget.  My budget is going to be around $15,000 adjusted for inflation.
  2. A goal statement or what you want to achieve that makes the project a successful learning experience.
  3. design goals.  The must haves to achieve the goal you want.
  4. If you want to get really crazy, an actual project plan.

We actually plan to create multiple prototypes, as our skills increase, so will the quality and “coolness” of our design.  This RC car we plan to use for our second prototype:

IMG_0849

The body of the prototype II car will a Porsche.  Prototype I is almost done 🙂 !

IMG_0851

The internal frame of prototype II car.  Once completed, it will have cameras, computers, and motors.  There will be other devices as well to help with autonomy.

Parent’s Guide to Designing and Building a Self-Driving Car with Their Kids – Part 2

“The two biggest challenges with this project is: 1) Getting your kids more interested in the project over their video games. 2) Convincing your child’s science teacher they did most of the work for the school science fair”

Before I start delving into the technical rigors of building a self-driving car.  I want to talk about kids.  As parents, we want our kids to be excited about things we get excited about.  When I thought up this project, I realized it was above what a fifth grader could do.  A self-driving car involves a multitude of technical subjects:  Advanced calculus, statistics, probability, linear algebra, deep learning, machine learning, electronics, computer science, telemetry, data science, mechanics, physics,…These are subjects kids are not expected to be good at.  But, I thought to myself  “It’s about the Journey…”.

As a parent, I want to expose my kids to STEM (Science, Technology, Engineering, Math).  This project is STEM³, meaning, it’s several factors above what is typically taught in grade school level STEM curriculum.  So how do you incorporate this into a child’s STEM education without frustrating your child and yourself in the process?

Divide things up into simpler lessons

So when my child and I started thinking about building a self-driving car, we started small.  We converted a radio controlled car into a self driving robot using the following components:

  1. Raspberry Pi 3 microcomputer.
  2. Arduino Microcontroller
  3. OpenCV computer vision software.
  4. Chassis of an old RC car.
  5. Electric 9V Motor

We then said, “What are the major components that a self-driving car would need”.  What we listed were:

  1. A Motor
  2. Steering
  3. A computer
  4. A camera

Doing our research into companies like Waymo, Google, Volvo, Tesla among others who are investing millions into autonomous technology, we began learning that among these pioneering companies are a community of tinkerers who are using open source code and open hardware to build autonomous RC cars.  Many of whom are blogging about it.  To learn more about these communities, I recommend the blog series. Becoming Human AI.  

With then focused on specific topics, that children could research and learn about.

  1. For the Motor: Pulse Width Modification.
  2. For the Steering:  Controlling sweeper servo moters.
  3. For the Computer:  Programming Raspberry PI with Python.
  4. For the Camera:  Using OpenCV for image processing.

School Science Fair Skepticism

When we attended my child’s science fair, we had spent weeks going over Pulse Width Modification, which is a way to control an electronic motor speed and building a sweeper motor for our prototype RC autonomous car.  We stuck on a Raspberry PI 3 computer which is basically a very cheep microcomputer that you can program and load up with an open source software package called OpenCV.  OpenCV  can detect images from a camera be recognize what that object is at least detect things in an image.  When we were done our science fair project looked like this:

 

IMG_2630[1]

We spent weeks putting this together, and I made certain my child understood each component in the car and had the knowledge to talk about it.  What I quickly noticed was among the baking soda volcanoes, and the dyed flower petal experiments, was a lot of skepticism that a fifth grader could put something like a “self driving robot”-thingy  together.

My child put the poster together and did all the calculations as I stood by and asked “So what can you include from those findings?”  The scientific method, which is the most important tool in science, was reiterated throughout the experiment:

  1. What are observations?
  2. What is your hypothesis?
  3. What methodology did you use?
  4. What is your experiment?
  5. What were you conclusions (what did you learn)?

This is what needs to be the basis for a child’s work in STEM projects in order for him or her to learn from the successes and failures of doing science and technology.

IMG_2635[1]

The picture above is my child’s science fair poster.  We worked pretty hard on it.  But my child did all of analysis and calculations and took all of the notes and typed it up.  I gave him a test to make certain that he understood everything.  The goal wasn’t to win (he didn’t) it was to get him interested in science and technology and show him that there are others that are excited as well…And many people were!

 

 

 

Parent’s Guide to Designing and Building a Self-Driving Car with Their Kids – Part 1

I was never really a “car” guy.  I always saw them as transportation from point A to point B.  I’ve abhorred walking into a car dealership to negotiate car purchases; and as a parent, keeping my car clean seemed like an constant battle.  Driving long distances, even if the car was clean, was especially arduous for me.  America is a huge country, so driving from my home in North Carolina to South Carolina or Tennessee is a five to seven hour odyssey without any of reward or enlightenment.

However, I was always enjoyed NASCAR (being from North Carolina), and I enjoyed watching YouTube videos of DIY rebuilders who take wrecks of cars and give them a new life.  So when I first learned about self-driving cars and the amount of tech and data that goes into making these autonomous machines, I started to take notice…

In this series, I will be discussing all the successes, failures and lessons learned building a life-size self-driving car…with my sons.  Subscribe and follow me on my blog to see how we’re doing.  I suspect it will be years before we are done (or give up) with it all.  It’ll be a lesson on how to build it, but also how to get your kids excited by a challenging STEM project.

Stay Tuned!!

 

 

 

 

 

Leveraging Memory in PL/SQL for Big Data Applications

When writing PL/SQL code for Big Data applications it’s always a good idea to leverage the database as much as possible.  This is particularly true for ETL or warehousing processing.

In my experience, I’ve tried to quantify objects in Oracle programming that work best to temporarily  store data.  As part of your Big Data application, anytime you need to leverage relational databases, here is a table that I typically use as a guideline for storing data for loading to or extracting data from a database.

Typically, I judge the what type of cursor object to used based on factors such as performance, the number of rows that need to be read, the operation that I am performing and how much memory that I would like to leverage.

Number of Rows Explicit Cursor  Temporary Table PL/SQL Memory Collections Record Collections  Typical Memory Used
<= 1000 check_mark7 check_mark7 10 MB
1000 to 500,000 check_mark7 check_mark7 check_mark7 100-500 MB
500,00 to 2,000,000+ check_mark7 check_mark7 500MB +

Although, explicit cursors can be used for much larger data sizes, if performance is your goal, a few thousand rows are optimal to maintain good performance, especially if you are opening and closing many cursors in a small window of time.