Відео

Sorry! I'm hoping to make more videos someday... but yeah, it was much easier to find time for this sort of thing in grad school.

Thanks! There isn't a template, but you can download the source code (SVG + HTML) for the presentation using the link in the description. It is written in Inkscape, with animations using Sozi.

Good question! I actually don't know any off the top of my head. I would start by looking into the references of the references at the end of this video. Let me know (or reply here) if you find a good one!

@@MatthewKelly2 sure thank you for the response

The "closed loop optimal control" problem is traditionally set up by solving the "Hamilton Jacobi Bellman" HJB equation numerically. This is only practical for low-dimensional problems due to the "curse of dimensionality". More recently, there has been a ton of progress made in approximate solutions to this problem using reinforcement learning. There are lots of good resources on that, and I'm not a real expert on RL. The most impressive work that I've seen there is by the Robot System Lab at ETH Zurich, where they used it to control a quadruped robot walking over rough terrain. More details here: rsl.ethz.ch/research/researchtopics/rl-robotics.html

@@MatthewKelly2 Thanks for the link...I find your way of teaching very interesting ...

In general, you can add just about anything you want as a decision variable in a trajectory optimization problem. One common example would be slack variables for a constraint, or a parameter in the dynamics model. Specifically in my OptimTraj Matlab package there is some limitation on the types of decision variables, which I did mostly to keep the user interface simple enough to use in an educational context. The GPOPS-II optimization package (also in Matlab) has a bit more flexibility, for example, allowing model parameters to be decision variables. I'm sure other packages have more flexibility as well. Or you could code up your own transcription, and then the sky is the limit. The key thing to keep in mind is that you will need to be careful that you still have a "well behaved" problem. For example, you'll still need consistent / continuous gradients of those new decision variables.

That sounds like an interesting problem to work on and learn from. I recommend starting simple and then adding constraints until you get to solving the full problem. For multi-phase references, look up the GPOPS-II user manual and associated papers by Anil Rao et al. There are definitely others out there, I just don't know them off the top of my head. The big picture concept with phase constraints is that you are connecting multiple "mini trajectory optimization problems" into one larger problem using "linkage constraints". The linkage constraints connect the boundary state on one phase to the boundary state of another phase. Interestingly, there is nothing that requires them to be sequential -- in general they can form a graph. The main challenge comes in setting up a clean user interface for specifying them, which is why they are not included in OptimTraj -- I wanted to keep it as simple as possible to use. That being said, they are not too challenging to hack into that software for a specific case. One side note -- my OptimTraj package is "optimized" for being easy to read and use as a learning tool, while still being fast enough to use for toy problems. The problem you are setting up is complicated enough that you are probably going to start running into some of the limitations of both OptimTraj and Matlab itself. If you're tied to Matlab, then I suggest switching to GPOPS-II, which is heavily optimized for performance on larger multi-phase problems. If you're not tied to Matlab, then I would suggest hunting around for a C++ implementation (perhaps with python bindings, if they exist) that would be faster.

It is a Benge 165F

@@MatthewKelly2 Would you recommend this brand or something of higher quality and durability. I have a moz tenor that doesn’t sound that great and the triggers keep getting stuck.

@@keithshockley3443 - I like this horn, but it is definitely "intermediate level". The trigger is reliable, but makes a bit of noise. The tolerances on the tuning slides are not great and they tend to get jammed if you're not careful. That being said, it sounds nice and is pretty indestructible. I got it in middle school and used it through high school. I got a much nicer Jupiter horn when I graduated. It was better in many ways, but I found the thayer valve continually got stuck and was hard to maintain. I don't play much anymore, so I eventually sold the Jupiter horn and keep the Benge around for playing occasionally. I would recommend the Benge horn for a beginner who is looking for an affordable upgrade from a student model, but not for someone who is looking to play at a higher level long term. For that I would recommend getting a slightly higher quality horn but but with the same type of valve as the Benge has. With better tolerances and the right padding on the stops the valve should be quiet and easy to maintain. The key features I would look for (apart from sounding good to you) would be nice tolerances on both the main and tuning slides. Good luck!

Mostly just different warm-up routines. The one “real” song near the end is a an excerpt from “simple gifts”. Glad that you like it!

There are some more-technical ways to show that the "integral of control effort squared" is a good objective function for second-order dynamical systems, but here is an intuitive explanation. The quadratic cost means that it is "expensive" to use a large control effort. Let's imagine that the controller wants to deliver a fixed impulse. If the cost was linear, then there would be no reason to prefer a 0.1s * 10N impulse vs a 1s * 1N impulse. With a quadratic cost the 1s * 10N impulse is "less expensive". Overall, this means that the "control-squared" objective tends to produce smooth solutions. This is important because the discrete approximation of the trajectory relies on the assumption that a polynomial spline can approximate the solution... which is only true if the solution is "smooth". It also happens that quadratic cost functions have really nice gradients, which helps the optimization numerically.

@@MatthewKelly2 Hi, Kelly, Thank you for your clear explanation, it's very helpful for me.

@@MatthewKelly2 Hi, Kelly, I also have another question, can we just understand the closed-loop solution is a global solution? If yes/no, can you give me simple reference material? Thank you very much.

@@yuelinzhurobotics - The type of trajectory optimization explained in this video can be thought of as a local solution, which could be evaluated to produce an open-loop controller. If you want the full global solution then you want a different technique. Look up "solving the Hamilton Jacobi Bellman (HJB) equations. You'll find that they become intractable for most non-trivial solutions. The general approach is to use reinforcement learning to approximate the global solution. This is an active area of research, and I'm not an expert -- lots of neat stuff to learn there. Another related thing is model-predictive control. That produces a local closed-loop controller (rather than open loop). The idea is that you continually solve a trajectory optimization (such as the cart-pole) swing up from the measured state, which then allows you to feed-back and stabilize errors. This is difficult to implement in practice because you need to solve the optimization very quickly (in real time, in your control loop). That being said, it can be done, once you get all of the details right.

Excellent question. Yes, you can make the duration of the trajectory a decision variable, and in many cases that allows the solver to find a "more optimal" solution. One detail is that the total duration then couples nearly every constraint in the optimization problem, making the overall problem more challenging to solve. You can also end up with some counter-intuitive behaviors if you don't set up the objective function and constraints carefully. One related topic: it is generally a bad idea to make the duration of all segments into decision variables, as the solver will often get stuck (Bett's discusses this in his book in some detail). Generally you fix the "mesh fraction" allocated to each segment, and then allow a small number of "phase durations" to be decision variables (see GPOPS-II documentation for more on this).

You can add a constraint on an intermediate point in either method, although the gradients are much nicer in the direct collocation formulation. The approach is to select one or more knot points at which to apply the constraint, and then append them to the optimization problem. You always need to watch out for "tunneling". For example, if you have widely spaced collocation points, you can compute a "feasible" trajectory that passes directly through a thin wall if each of the collocation points is "not in collision" with the wall. There are various heuristics for avoiding this, most either based on "padding" the constraint (making the wall wider than it really is) or adding extra collocation points near obstacles.

@@MatthewKelly2 thanks for the answer! I am thinking on how to implement this intermediate constraint though. Maybe using a conditional statement (in the tree example one could do something like this: IF x=obstacle_coordinate AND y<= tree_height THEN Ceq=1 ) ? It doesn't seem very efficient to me though..

@@emanuel4516 - Correct. The 'if' statement would be no good, as it causes a discontinuity in the sparsity pattern of the constraint jacobian. You'll want to add a constraint something like: 'distance_to_obstacle(state(i)) > 0', and then apply that that constraint for every single point on the trajectory that could come in contact with the tree. This allows the NLP solver itself to figure out if the constraint is active or not. The 'distance_to_obstacle' function should be continuous (no if statements).

Yes, although the details start to get complicated. The general idea is to add a distance-squared constraint at each collocation point. You need to be somewhat careful about avoiding "tunneling". Depending on the complexity of your kinematics, these additional constraints can be expensive to evaluate. I can't think of any good references off the top of my head, but I'm sure that they're out there.

If you follow the first link (www.matthewpeterkelly.com/tutorials/trajectoryOptimization/cartPoleCollocation.svg), you'll discover a big SVG file that has all of the slides in it. I created this file using Inkscape. There is a LaTeX plugin for inkscape, although it was pretty unreliable at the time of writing this presentation. The animations (really just links to different views onto each slide) were auto-generated using Sozi, and animation front-end for Inkscape.

Sorry - I don't have much experience using Scipy for trajectory optimization! Let me know if you find a good library.

You're welcome to check out the Python package I've been working on, Pycollo. The GitHub is here: github.com/brocksam/pycollo. It's a work in progress but it can quickly and efficiently solve this problem among many others (in fact, this problem is implemented as an example)! I'm working on making it conda-installable right now.

@@MatthewKelly2 No problem!

@@sambrothie Wow thank you this is very neat as well. Currently I am also trying to solve it a longer way so I can better learn about how this all works. Your package is a great tool for me to check. Do you happen to know what the equality constraints (final/initial state) would look like if I were to use scipy minimize?

Good question, if I'm understanding it correctly. Let me try restating it: How can we use a non-smooth function to represent the trajectory in a nonlinear program that requires everything to be smooth? The key here is that the trajectory (piecewise linear in time) is discontinuous in time. The NLP doesn't care at all about time: it only cares about decision variables. If you carefully look at the gradients, you'll find that the gradient of the trapezoidal discretion with respect to the decision variables is continuous. Another possible interpretation: What if I need the solution trajectory to be continuous with respect to time? Then you can use a higher-order method that has a solution trajectory that is continuous with respect to time.

@@MatthewKelly2 this actually makes a lot of sense, the NLP solver being independent of the time. Thanks once again for the attention! Learning a lot from!!

It depends what you mean by "input law is discontinuous". Let's be more specific. If the objective or constraint function has a discontinuity, then you've got trouble. This will cause a jump in the gradients that will break most continuous optimization solvers. The general "trick" is to reformulate your problem in such a way as to inform the solver about the discontinuity and let it manage it via switching the set of active constraints. This is often done using slack variables. A common example is to replace an absolute value function with two positive slack variables and an equality constraint, as is described in Bett's book (see references at end of presentation or in my journal paper linked in the description). I If the discontinuity occurs in the solution of the trajectory optimization, which can happen for some smooth optimization problems, then you can also have problems. The NLP will often converge, but your discretization will often be bad. The solution here is to iteratively adjust your mesh, this can be done manually, or through adaptive meshing. See the research by Anil Rao et al. on HP adaptive meshing that is used in GPOPS for a concrete example and thorough analysis.

@@MatthewKelly2 thank you so much for your complete answer!! However, I don’t know if I’m mixing things here, but the slack variables is part of the KKT conditions for optimality, right? And what if I cannot reformulate my NLP to make it continuous for my feasible set, this means that I’ll have to go for stochastic methods of optimization? Sorry for coming with more questions....

@@pedrocalorio1655 Some NLP solvers use slack variables internally, but you don't need to worry about that. The slack variables that I'm talking about would be added directly as decision variables while formulating the NLP, typically as extra state or control variables. The `minimumWork/MAIN_cstWork.m` example in my OptimTraj software shows a simple example of using slack variables to represent an absolute value function. Similar tricks can be used for min() and max(). There are also various ways to "smooth out" functions and interpolation from tables. Iterative methods can be made "smooth" by using a fixed number of iterations. You can use stochastic optimization, but I've never gotten that to work reliably. The problem is that trajectory optimization requires a large number of equality constraints for the defects, and stochastic optimizer have a really hard time with equality constraints. Put differently, the whole idea behind collocation methods (like I describe here) is to put the optimization into a format the makes it easy for sparse gradient-based solvers to optimize. If you switch to a stochastic solver, then you would probably want a different formulation entirely, something closer to single shooting (which has its own set of challenges).

Good question! Knot points are the points along the trajectory that separate continuous polynomial sections. Put differently, they are the places where there is a discontinuity in some derivative. This terminology is more broad than trajectory optimization: it comes from polynomial splines. If the spline has N segments, then it will have N+1 knot points, including the boundaries of the spline. Collocation points are those points where the system dynamics are evaluated, and represent the points that are used to discretize the system dynamics. The "collocation error" will be zero at these points for a valid solution to the NLP. The collocation and knot points often overlap, especially for low-order methods. For example, in trapezoidal collocation, the knot and collocation points are identical. In backwards euler discretization, all collocation points overlap with knot points, but there is a single knot point (the first point on the trajectory), that is not a collocation point. For the hermite-simpson method, all knot points overlap with collocation points, but there is an extra collocation point in the middle of each segment that does not overlap with a knot point.

@@MatthewKelly2 That clears my doubt. Thank you!

This animation is playing back the solution to a trajectory optimization problem. If you follow the "source code" link in the description it will bring you to OptimTraj, a trajectory optimization library that I wrote which includes this problem as an example. There is also a link to a tutorial that explains some of the concepts behind trajectory optimization. You can definitely make a LQR controller stabilize this sort of trajectory. The trick is to linearize around the trajectory, not the final solution. Check out Russ Tedrake's Underactuated Robotics class at MIT, where he discusses this topic in depth: underactuated.mit.edu/lqr.html#example2

​@@MatthewKelly2 Thank you much for your information!

@@MatthewKelly2 Thank you so much again. Until now I didn't know what I have to study. Thank you very much for introducing about this field.

Good question! In the demo that I wrote, it is a hard-coded assumption that the pole length is fixed. That being said, you can definitely make a version of the dynamics function that supports a time-varying pole length.There are three ways that I could imagine doing it. (1) Add a passive element, such as a spring, that applies along the pole and allows the mass to move along the length. This would require writing out a new dynamics equation, but would not require adding new inputs to the function. (2) Explicitly make the length of the pole a function of time and then pass that function in as a parameter. Then take the time that is passed into the dynamics function and use it to evaluate the pole length (and its derivatives). Then you need to update the dynamics to properly account for accelerations related to changes in pole length. (3) Add an actuator that can control the length of the pole (easiest to use a force actuator here), and then update the dynamics accordingly. This would allow the optimization to "select" the correct length for the pole. In all cases you would need to update the dynamics of the system, but that shouldn't be too tricky as it is a relatively simple system. Good luck!

Good question! In the demo that I wrote, it is a hard-coded assumption that the pole length is fixed. That being said, you can definitely make a version of the dynamics function that supports a time-varying pole length.There are three ways that I could imagine doing it. (1) Add a passive element, such as a spring, that applies along the pole and allows the mass to move along the length. This would require writing out a new dynamics equation, but would not require adding new inputs to the function. (2) Explicitly make the length of the pole a function of time and then pass that function in as a parameter. Then take the time that is passed into the dynamics function and use it to evaluate the pole length (and its derivatives). Then you need to update the dynamics to properly account for accelerations related to changes in pole length. (3) Add an actuator that can control the length of the pole (easiest to use a force actuator here), and then update the dynamics accordingly. This would allow the optimization to "select" the correct length for the pole. In all cases you would need to update the dynamics of the system, but that shouldn't be too tricky as it is a relatively simple system. Good luck!

@@MatthewKelly2 I will try to tinker with these ideas. Thank you so much for your suggestions!

ua-cam.com/video/fRyUf-GY754/v-deo.html

not really

I believe so. A "Trajectory Optimization" problem is just solving a special case of the Hamilton Jacobi Bellman (HJB) equations for a single point. In fact, you can use the full solution (optimal policy) from the HJB equations to reconstruct an optimal trajectory. The problem is the "curse of dimensionality" -- usually it is impractical to solve the HJB directly, but it is feasible to compute a locally optimal solution at a single point (trajectory optimization). I believe you could derive any trajectory optimization framework from the HJB equations, but I'm less familiar with the mathematics involved, and find the derivations in this video easier to understand. You can definitely set up and solve trajectory optimization starting from the Pontryagin's maximum principle, and that is closely related to how "indirect" methods for trajectory optimization work. I know less about those techniques, but if you follow the references at the end of the presentation you'll be able to learn more.

@@MatthewKelly2 Thank you so much for taking the time to answer! I really have a better understanding now... good point about the Pontryagin's max principle and the "indirect" methods indeed.

ua-cam.com/video/fRyUf-GY754/v-deo.html