Skip to content
This repository was archived by the owner on Jan 29, 2026. It is now read-only.

adding archive note #201

Open
jamaya2001 wants to merge 88 commits into
base: master
Choose a base branch
from
Open

adding archive note #201

jamaya2001 wants to merge 88 commits into from

Conversation

@jamaya2001
Copy link
Contributor

@jamaya2001 jamaya2001 commented Dec 9, 2022

Pattern is being retired from the IBM Developer site, so adding archive note to the GH repo.

@Tomcli Can you approve/merge?

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

whummer and others added 30 commits February 9, 2018 19:41
@whummer
Merge pull request IBM#2 from whummer/fix/travis-img
b5b9868 
[don't merge yet] Fix package names and travis config
@animeshsingh @whummer
Update README.md (IBM#4)
32042e7
@animeshsingh
Adding paper references - changing main title (IBM#5)
a9167d2
@animeshsingh @Tomcli
Removing the extra bracket around references (IBM#7)
209572a
@Tomcli @whummer
Update instructions with Cloud Object Storage (IBM#3)
7057c84 
Update instructions with Cloud Object Storage
@animeshsingh @Tomcli
Architecture diagram + adjusted specs for memory and cpu (IBM#12)
85ef00c 
* arch diag

* arch diag

* Update README.md

* adding specs

* adding specs
@Tomcli @animeshsingh
Add support to none VM_TYPE Clusters (IBM#15)
2fc8966 
* update none VM_TYPE

* polish export commands
@Tomcli @whummer
add user guide (IBM#10)
51b6e58
@Tomcli @whummer
* Fix Caffe Example (IBM#19)
2dc0b34 
* Extent make test-submit waiting time
@Tomcli @animeshsingh
Assign default edit role to lcm to support kubernetes 1.9 and above (I…
8581fec 
…BM#22)

* assign default edit role to lcm

* add helm value options for 1.7 and below
@animeshsingh @Tomcli
Some changes in Prereqs + Pointers to ICP (IBM#23)
7749f82 
* adding prereqs and bumping user guide in front

* adding prereqs and bumping user guide in front
@Tomcli @animeshsingh
Don`t Merge Yet - Add Caffe2, PyTorch and GPU support(using Accelerat…
80769d0 
…ors) (IBM#24)

* add caffe2 and pytorch cpu support

* update LCM, learner config file, and example jobs

* fix pytorch example bug

* Update gpu-guide.md

* Update gpu-guide.md

* merge CPU and GPU examples into a single example

* add more tf framework versions

* fix typo

* add S3 prereq
@Tomcli @animeshsingh
Update UI instructions to manually create model zip files (IBM#29)
008655a 
* update UI instructions

* fix command
@animeshsingh @Tomcli
revised architecture details (IBM#30)
d8815a7
@Tomcli @animeshsingh
Update helmignore to avoid oversized helm configmap (IBM#32)
4c469e0
@Tomcli @animeshsingh
update UI dependencies and add font-awesome (IBM#33)
9796931
@whummer @animeshsingh
add initial list of contributors (IBM#36)
255213b
@animeshsingh @Tomcli
Adding Contributors in CONTRIBUTING.MD (IBM#37)
e083b80 
* adding contributors

* Update README.md
@animeshsingh @Tomcli
Updating maintainers file (IBM#38)
4434e2d 
* Updating maintainers file

* Update MAINTAINERS.md
@Tomcli @animeshsingh
Add converting script for FfDL to WML and vice versa (IBM#39)
ebba9d1 
* add converting script

* update converter readme and update tensorflow version

* update troubleshooting

* Update README.md

* Update gpu-guide.md

* Update README.md

* Update README.md

* Update README.md
@animeshsingh
Linkage to watson studio (IBM#40)
9365a16 
* Adding references to Watson Studio

* Update README.md

* Rename README.md to ffdl-wml.md

* Update README.md

* Create train-deploy-wml.md

* Update train-deploy-wml.md

* Update README.md

* Update ffdl-wml.md

* Update ffdl-wml.md

* Update ffdl-wml.md

* Update README.md

* Update README.md

* update WML instructions

* revert tf example

* update caffe manifest
@Tomcli @animeshsingh
Remove Vagrant reference (IBM#43)
35e549a
@Tomcli @animeshsingh
update tensorflow example (IBM#44)
25bd031
@Tomcli @animeshsingh
Temporary fix for CI with k8s 1.9.4 (IBM#45)
5ab24e4 
* Update feature-gates for k8s 1.9.4 and above

* Update troubleshooting

* Update README.md
@Tomcli @sboagibm
Update learner entrypoint command to work on k8s 1.9.4 and above (IBM#47
609fbce 
)

* update learner entrypoint command

* update troubleshooting

* Update train.sh path to match with DLaaS.
@animeshsingh @Tomcli
Removing orphan vagrant references (IBM#50)
7abbd1f
@Tomcli @animeshsingh
Add developer guide and some customizable features for development bu…
6a41a84 
…ild (IBM#51)

* * Add codebase configuration for device plugin and custom learner images
* Add developer guide for those who want to do a custom FfDL build

* update developer-guide

* fix declare type
@kant @animeshsingh
Minor fix on #273 & #370 (IBM#35)
0cb9935 
Plus minor fixes
@animeshsingh @Tomcli
Creating CLA (IBM#52)
6281c12 
* Creating CLA

* Update CLA.md

* Update CONTRIBUTING.md
@Tomcli @animeshsingh
fix logging bug (IBM#53)
3545032
ckadner and others added 29 commits July 5, 2018 11:54
@ckadner @sboagibm
Remove 4 minute timeout for log follow process (IBM#107)
b339cbe 
* Remove 4 minute timeout for log follow process (IBM#106)

The process that follows the training logs of an ongoing training job
should not timeout after 4 minutes. Instead the log follow process
should complete after the training job itself is finished.

This behavior is necessary to enable chaining up commands to create
machine learning pipelines, where subsequent commands require the output
data of the training job whose logs are being "followed" like in our
ART notebook.

This commit reinstates the log follow behavior prior merge of PR IBM#79

* Updates suggested by sboagibm

Intention was to not rely on a long term stream being held open, 
but to be able to re-open a new stream starting from where the 
old left off, if the connection terminates.
@ckadner @animeshsingh
Update ART notebook(s) after PR IBM#79 (IBM#116)
cbc8da9 
* Update ART Notebook after PR IBM#79

- Load cluster configuration from environment variables
- Require PUBLIC_IP and KUBECONFIG instead of CLUSTER_NAME and VM_TYPE
- Use storage type "mount_cos" (s3fs) instead of "s3_datastore"

* Update ART demo notebook after PR IBM#79

- Load cluster configuration from environment variables
- Require PUBLIC_IP and KUBECONFIG instead of CLUSTER_NAME and VM_TYPE
- Use storage type "mount_cos" (s3fs) instead of "s3_datastore"
@ukclivecox @animeshsingh
updates for seldon-core 0.2 (IBM#118)
0202204
@Tomcli @animeshsingh
Update deprecated framework versions (IBM#119)
4833125 
* update dl framework versions

* update examples with new framework tags
@Tomcli @animeshsingh
Update Fashion MNIST example with Seldon-core 0.2 (IBM#120)
0b93b0a 
* update fashion mnist example with seldon 0.2

* fix readme
@sboagibm
Sboag dlaas pull june14 (IBM#97)
0f053c1 
* Pointed travis testing to do hostmount minikube

* Debugging permissions error.

* Fix to mkdir problems.

* Fixed Makefile syntax.

* Printing debugging information about pods.

* Printing debugging information about pods.

* Printing debugging information about pods.

* Printing debugging information incl kubectl get pod.

* Enabled debug mode.

* Again.

* Set debug as default.

* tracing from the trainer to lcm

* more debugging

* added lower level logging

* dist: xenial

* Update .travis.yml

* fix typo

* Trying to fix Travis issue.

* Fixed Travis issue.

* Followed Tommy's request and increased resource limits to values from before. Might break CI.

* Parameterized memory values like Tommy requested.

* Attempt to fix CI.

* Removed excessive debug statements and cleaned comments. Probably breaks code.

* DLaaS pull june 14, with security mods

* fixed glide problem

* Added Image.go etc. files, deleted learner_test.go

* temporarily disable framework validation

* FIXME: Disable validation check for bucket until conditionalize for s3fs vs.  option.

* fixed two bugs related to volume mounting

* I think mostly just logging changes

* basic success

* Add FfDL.iml to .gitignore

* removed docker ref to csf_env.properties

* Test for mount_cos before attempting s3 validation

* fixed hostmount by pre-setup of model code in Makefile

* fixed missing import

* log HELM_DEPLOY_DIR, add a bunch of logging for the ci test

* Added create-volumes to jenkins file, more verbose docker build for ui

* Wound back Angular to 6.0.8

* Quiet docker-build-ui docker build

* merged bin/create_static_volumes_config2.sh into bin/create_static_volumes_config.sh
@Tomcli @sboagibm
Version control for prebuilt images (IBM#122)
cc7388a 
* update prebuild image version, update helm chart to 0.1.1

* fix make deploy bug
@Tomcli @fplk
Make helm charts and scripts compatible to deploy FfDL on any namespa…
ccce821 
…ce (IBM#110)

* make helm charts and scripts compatible to deploy FfDL on any namespace

* allow users to export all the enviornment variables in a txt file

* Update readme with new notice

* Fix typo

* Update static volumes config v2 namespace parameter

* capitalize NAMESPACE, update Makefile, developer guide, and trobleshooting.
@Tomcli @fplk
Fix NAMESPACE typo on one of the deployment script (IBM#126)
8fbae4f
@Tomcli @animeshsingh
use python zipfile instead of bash unzip (IBM#128)
f5dcaf6
@ckadner @Tomcli
ART notebook did not load due to missing comma (IBM#129)
e358a42
@Tomcli @fplk
update statsd flags with double dashes (IBM#132)
9e19e83 
LGTM. Ran fine / fixed statsd issue on Ubuntu 18.04 Vagrant VM.
@animeshsingh @Tomcli
Simplifying README layout (IBM#135)
937c323 
* Simplifying README

* Simplifying README

* Create detailed-installation-instructions.md

* Update README.md

* Update detailed-installation-instructions.md

* Update and rename detailed-installation-instructions.md to detailed-installation-guide.md

* Update detailed-installation-guide.md

* Update detailed-installation-guide.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md
@Tomcli @animeshsingh
Pytorch 1.0 patch (IBM#137)
cc1deed 
* added pytorch distributed example draft

* use pytorch community image

* added experimental pytorch example

* update launch example

* add onnx example

* update c10d dist example

* update cuda method for gpu and group definition for c10d

* update c10d to based on SEP 11 build on PyTorch master.

* update custom pytorch version name

* update data parallelism examples

* update distributed training examples

* update pytorch c10d examples

* delete dummy file

* fix minor bugs

* update example to sep 25 build

* update readme

* update changes for distributed CPU job

* update multi-gpu code

* update multi-gpu code

* update world_size for multi-gpu senario

* update world_size for multi-gpu senario

* add seldon ngraph example, update data parallelism with multi gpu

* remove unnecessary device code

* added pytorch mpi core changes and seldon readme

* added pytorch mpi core changes and seldon readme

* update readme

* update example readme and remove old distributed example

* update c10d-paralleism example naming and readme.

* update readme with better naming and consistancy
@animeshsingh @Tomcli
PyTorch and ONNX support (IBM#138)
5985685 
* Create PyTorch.md

* Update PyTorch.md

* Update PyTorch.md

* Update PyTorch.md

* Update PyTorch.md

* arch-image

* Update PyTorch.md

* Adding temporary linkage to PyTorch 1.0

* Fix broken link for Seldon example

* Correcting Horovod naming
@kant @animeshsingh
Minor proposals on paragraph IBM#57 (IBM#140)
142f31d 
Markdown: ¿loose on paragraph format?
@ckadner @Tomcli
Fix Firefox file upload error in Fashion demo (IBM#142)
d18fbca
@animeshsingh
Travis CI: lint Python for syntax errors and undefined names (IBM#147)
d0c8e70 
* Travis CI: lint Python for syntax errors and undefined names

In Travis CI, add a Python linting step that runs [flake8](http://flake8.pycqa.org) to find syntax errors and undefined names.

[flake8](http://flake8.pycqa.org) testing of https://github.com/IBM/FfDL on Python 3.7.0

$ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__
```
./etc/examples/c10d-onnx-mpi/model-files/train_dist_onnx_mpi.py:93:55: F821 undefined name 'bsz'
    num_batches = ceil(len(train_set.dataset) / float(bsz))
                                                      ^
./etc/examples/c10d-dist-onnx/model-files/train_dist_onnx.py:121:14: E999 SyntaxError: positional argument follows keyword argument
              ', epoch ', epoch, '. avg_loss: ',
             ^
1     E999 SyntaxError: positional argument follows keyword argument
1     F821 undefined name 'bsz'
2
```

__E901,E999,F821,F822,F823__ are the "_showstopper_" flake8 issues that can halt the runtime with a SyntaxError, NameError, etc. Most other flake8 issues are merely "style violations" -- useful for readability but they do not effect runtime safety.
* F821: undefined name `name`
* F822: undefined name `name` in `__all__`
* F823: local variable name referenced before assignment
* E901: SyntaxError or IndentationError
* E999: SyntaxError -- failed to compile a file into an Abstract Syntax Tree

* Undefined name: bsz --> batch_size

* Fix syntax error: print() does not accept an 'rank=' parameter
@Tomcli
Update dind port forward script to be compatible with K8S client 1.9.…
abd8294 
…x and 1.8.x (IBM#150)
@Tomcli @animeshsingh
Update converter bx commands to ibmcloud (IBM#156)
43f136a 
* update converter bx commands to ibmcloud

* update converter bx commands to ibmcloud
@animeshsingh @Tomcli
Removing references to an internal notebook (IBM#157)
dc84caa 
Fix for IBM#155
@fplk
Fix dind (IBM#158)
6824501 
* Updated setup script to K8s 1.13.

* Modified Makefile
@Tomcli @animeshsingh
Update detail installation instructions on DIND to reflect the change…
56067d6 
…s in IBM#158 (IBM#159)

* Update detailed-installation-guide.md

* Fix links and typos. Add explanation on the manifest fields.
@erjanmx @animeshsingh
Fix typo (IBM#163)
f6afd4c
@animeshsingh
Removing 'latest' reference (IBM#170)
bf6708d
@imgbot @animeshsingh
[ImgBot] Optimize images (IBM#172)
b1cb3b4 
*Total -- 4,710.69kb -> 3,241.39kb (31.19%)

/dashboard/src/assets/img/ffdl-blue.png -- 6.10kb -> 3.07kb (49.66%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/ffdl-blue.png -- 6.10kb -> 3.07kb (49.66%)
/dashboard/src/assets/img/ffdl-text.png -- 10.97kb -> 5.73kb (47.73%)
/dashboard/src/assets/img/ffdl.png -- 6.04kb -> 3.31kb (45.14%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/p1.png -- 488.27kb -> 275.07kb (43.67%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/ui-example.png -- 80.93kb -> 47.47kb (41.35%)
/docs/images/ui-example.png -- 80.93kb -> 47.47kb (41.35%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/ffdl-fashion.png -- 301.63kb -> 177.72kb (41.08%)
/docs/images/ffdl-architecture.png -- 529.86kb -> 314.10kb (40.72%)
/docs/images/ffdl-pattern-arch.png -- 413.11kb -> 246.18kb (40.41%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/fashion-arch.png -- 444.84kb -> 266.69kb (40.05%)
/docs/images/horovod.png -- 73.13kb -> 44.93kb (38.56%)
/demos/fashion-mnist-adversarial/images/ffdl-art-jupyter.png -- 258.40kb -> 160.20kb (38%)
/community/FfDL-H2Oai/images/ffdl-h203.png -- 246.33kb -> 171.45kb (30.4%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/images/1.jpg -- 308.56kb -> 221.44kb (28.23%)
/etc/examples/images/pytorch-ffdl-onnx.png -- 277.25kb -> 199.65kb (27.99%)
/demos/fashion-mnist-training/sample-test-data/sneaker.jpg -- 29.37kb -> 24.93kb (15.13%)
/demos/fashion-mnist-training/sample-test-data/trouser.jpg -- 26.84kb -> 22.79kb (15.08%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/images/5.jpg -- 42.41kb -> 36.41kb (14.15%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/images/4.jpg -- 29.22kb -> 25.33kb (13.3%)
/demos/fashion-mnist-training/sample-test-data/sandal3.jpg -- 40.43kb -> 35.15kb (13.04%)
/docs/images/ffdl-arch-web.png -- 191.11kb -> 166.52kb (12.87%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/images/7.jpg -- 34.31kb -> 30.11kb (12.25%)
/demos/fashion-mnist-adversarial/images/adv_sample_predictions.png -- 142.53kb -> 125.72kb (11.79%)
/demos/fashion-mnist-training/sample-test-data/coatwhite.jpg -- 34.56kb -> 30.51kb (11.73%)
/demos/fashion-mnist-training/sample-test-data/sneakerbrown.jpg -- 46.75kb -> 41.43kb (11.37%)
/demos/fashion-mnist-training/sample-test-data/dress2.jpg -- 46.95kb -> 41.65kb (11.3%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/images/0.jpg -- 37.63kb -> 33.53kb (10.9%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/images/8.jpg -- 81.46kb -> 72.65kb (10.82%)
/demos/fashion-mnist-training/sample-test-data/trouser2.jpg -- 29.28kb -> 26.11kb (10.82%)
/demos/fashion-mnist-training/sample-test-data/redtshirt.jpg -- 34.92kb -> 31.46kb (9.91%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/images/3.png -- 2.68kb -> 2.44kb (8.98%)
/demos/fashion-mnist-training/sample-test-data/boot2.jpg -- 45.36kb -> 41.31kb (8.93%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/images/9.jpg -- 76.05kb -> 69.67kb (8.39%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/images/6.jpg -- 46.66kb -> 42.80kb (8.28%)
/demos/fashion-mnist-adversarial/images/ffdl.png -- 15.40kb -> 14.30kb (7.18%)
/demos/fashion-mnist-training/fashion-mnist-webapp/static/img/codait-logo.jpg -- 144.32kb -> 139.03kb (3.67%)
@spzala
Add copyright info (IBM#193)
c57f73b 
Adding name and year per when the file was added.

Signed-off-by: Sahdev Zala <spzala@us.ibm.com>

Signed-off-by: Sahdev Zala <spzala@us.ibm.com>
@Tomcli
Helm charts refactor(IBM#139)
0556361 
* refactor helm charts

* first draft of the new Travis CI script

* first draft of the new Travis CI script

* add helm init for CI

* enhance CI script

* fix typo in script

* package helm chart and move to docs repo for helm chart hosting preparation

* condense the 4 helm charts into 3

* update detailed installation guide

* update developer guide

* update helm chart naming
@jamaya2001
adding archive note
cea1853 
Pattern is being retired from the IBM Developer site, so adding archive note to the GH repo.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.